[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:25 dist-test-slave-znh6 krb5kdc[28254](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:25 dist-test-slave-znh6 krb5kdc[28254](info): set up 2 sockets
Aug 10 18:58:25 dist-test-slave-znh6 krb5kdc[28254](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 18:58:26.629091 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:26 dist-test-slave-znh6 krb5kdc[28254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:26.758376 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37955
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37955
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:26.870868 28274 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:26.871158 28274 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:26.871241 28274 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:26.875228 28274 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:26.875306 28274 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:26.875344 28274 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:26.875408 28274 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:26.875448 28274 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:26.880750 28274 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28274
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:26.881800 28274 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:26.882889 28274 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:26.892637 28283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:26.892692 28274 server_base.cc:1034] running on GCE node
W20260810 18:58:26.892644 28286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:26.892637 28282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:26.893458 28274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:26.894631 28274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:26.895800 28274 hybrid_clock.cc:648] HybridClock initialized: now 1786388306895790 us; error 30 us; skew 500 ppm
Aug 10 18:58:26 dist-test-slave-znh6 krb5kdc[28254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388306, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:26.899147 28274 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:26.900105 28274 webserver.cc:460] Webserver started at http://127.27.147.190:33467/ using document root <none> and password file <none>
I20260810 18:58:26.900699 28274 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:26.900763 28274 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:26.901026 28274 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:26.903508 28274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "e40a289c3f9b450ca3ca00f4b6096417"
format_stamp: "Formatted at 2026-08-10 18:58:26 on dist-test-slave-znh6"
server_key: "9d10179b69f856c61d4138dc8ca29ca4"
server_key_iv: "0fed30f7f0af58a8109459cb42f5397c"
server_key_version: "encryptionkey@0"
I20260810 18:58:26.904137 28274 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "e40a289c3f9b450ca3ca00f4b6096417"
format_stamp: "Formatted at 2026-08-10 18:58:26 on dist-test-slave-znh6"
server_key: "9d10179b69f856c61d4138dc8ca29ca4"
server_key_iv: "0fed30f7f0af58a8109459cb42f5397c"
server_key_version: "encryptionkey@0"
I20260810 18:58:26.907995 28274 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.000s sys 0.004s
I20260810 18:58:26.910408 28303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:26.911834 28274 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:26.911988 28274 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "e40a289c3f9b450ca3ca00f4b6096417"
format_stamp: "Formatted at 2026-08-10 18:58:26 on dist-test-slave-znh6"
server_key: "9d10179b69f856c61d4138dc8ca29ca4"
server_key_iv: "0fed30f7f0af58a8109459cb42f5397c"
server_key_version: "encryptionkey@0"
I20260810 18:58:26.912102 28274 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:26.925426 28274 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:26.928890 28274 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:26.929143 28274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:26.938457 28378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37955 every 8 connection(s)
I20260810 18:58:26.938480 28274 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37955
I20260810 18:58:26.939612 28274 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:26.942534 28379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:26.944962 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28274
I20260810 18:58:26.945124 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:26.946893 28238 external_mini_cluster.cc:1428] Setting key b73a3db143d27cec376b12f6a688b68e
I20260810 18:58:26.949863 28379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417: Bootstrap starting.
I20260810 18:58:26.952687 28379 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:26.953603 28379 log.cc:824] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:26.956065 28379 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417: No bootstrap required, opened a new log
I20260810 18:58:26.960548 28379 raft_consensus.cc:348] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40a289c3f9b450ca3ca00f4b6096417" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37955 } }
I20260810 18:58:26.960758 28379 raft_consensus.cc:374] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:26.960817 28379 raft_consensus.cc:729] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e40a289c3f9b450ca3ca00f4b6096417, State: Initialized, Role: FOLLOWER
I20260810 18:58:26.961259 28379 consensus_queue.cc:260] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40a289c3f9b450ca3ca00f4b6096417" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37955 } }
I20260810 18:58:26.961385 28379 raft_consensus.cc:388] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:26.961472 28379 raft_consensus.cc:482] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:26.961615 28379 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:26.962695 28379 raft_consensus.cc:504] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40a289c3f9b450ca3ca00f4b6096417" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37955 } }
I20260810 18:58:26.963074 28379 leader_election.cc:302] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e40a289c3f9b450ca3ca00f4b6096417; no voters:
I20260810 18:58:26.963438 28379 leader_election.cc:288] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:26.963554 28385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:26.963802 28385 raft_consensus.cc:686] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [term 1 LEADER]: Becoming Leader. State: Replica: e40a289c3f9b450ca3ca00f4b6096417, State: Running, Role: LEADER
I20260810 18:58:26.964196 28385 consensus_queue.cc:237] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40a289c3f9b450ca3ca00f4b6096417" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37955 } }
I20260810 18:58:26.964692 28379 sys_catalog.cc:564] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:26.965934 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e40a289c3f9b450ca3ca00f4b6096417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40a289c3f9b450ca3ca00f4b6096417" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37955 } } }
I20260810 18:58:26.966065 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:26.966305 28388 sys_catalog.cc:455] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e40a289c3f9b450ca3ca00f4b6096417. Latest consensus state: current_term: 1 leader_uuid: "e40a289c3f9b450ca3ca00f4b6096417" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e40a289c3f9b450ca3ca00f4b6096417" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37955 } } }
I20260810 18:58:26.966390 28388 sys_catalog.cc:458] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:26.966851 28399 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Aug 10 18:58:26 dist-test-slave-znh6 krb5kdc[28254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388306, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:26.970382 28399 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:26.974548 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:26.950881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34976 (local address 127.27.147.190:37955)
0810 18:58:26.951425 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:26.951450 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:26.952556 (+ 1106us) server_negotiation.cc:407] Connection header received
0810 18:58:26.956629 (+ 4073us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:26.956648 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:26.956961 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:26.957235 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:26.960179 (+ 2944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:26.961339 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:26.963147 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:26.963888 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:26.967781 (+ 3893us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:26.968995 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:26.969001 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:26.969012 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:26.969037 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:26.970882 (+ 1845us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:26.971440 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:26.971445 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:26.971449 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:26.971555 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:26.971939 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:26.971942 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:26.971943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:26.972438 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:26.972635 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:26.973336 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 18:58:26.973622 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":144,"threads_started":1}
I20260810 18:58:26.976523 28399 catalog_manager.cc:1347] Generated new cluster ID: 62f6986d8d9b4899959253bbfe618d07
I20260810 18:58:26.976583 28399 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:26.988803 28399 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:26.989935 28399 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:26.999454 28399 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e40a289c3f9b450ca3ca00f4b6096417: Generated new TSK 0
I20260810 18:58:27.000231 28399 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:27.035761 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37955
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:27.152839 28419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:27.153096 28419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:27.153172 28419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:27.157117 28419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:27.157168 28419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:27.157264 28419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:27.162470 28419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37955
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:27.163579 28419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:27.164752 28419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:27.174803 28434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:27.174847 28438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:27.174897 28419 server_base.cc:1034] running on GCE node
W20260810 18:58:27.174851 28431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:27.175453 28419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:27.176057 28419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:27.177217 28419 hybrid_clock.cc:648] HybridClock initialized: now 1786388307177204 us; error 25 us; skew 500 ppm
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:27.180037 28419 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:27.181869 28419 webserver.cc:460] Webserver started at http://127.27.147.129:45405/ using document root <none> and password file <none>
I20260810 18:58:27.182441 28419 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:27.182507 28419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:27.182750 28419 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:27.184780 28419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "9109d133bc7f4062acd98a409bb5e723"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "9758b5046e8bf429fde9725f29be4134"
server_key_iv: "f1d2342383432ce61af89eafc44e5c8d"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.185364 28419 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "9109d133bc7f4062acd98a409bb5e723"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "9758b5046e8bf429fde9725f29be4134"
server_key_iv: "f1d2342383432ce61af89eafc44e5c8d"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.188992 28419 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:58:27.191401 28452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.192579 28419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:27.192703 28419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "9109d133bc7f4062acd98a409bb5e723"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "9758b5046e8bf429fde9725f29be4134"
server_key_iv: "f1d2342383432ce61af89eafc44e5c8d"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.192807 28419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:27.219390 28419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:27.222880 28419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:27.223119 28419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:27.223791 28419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:27.224680 28419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:27.224756 28419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.224835 28419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:27.224886 28419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.234917 28419 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:41309
I20260810 18:58:27.234936 28622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:41309 every 8 connection(s)
I20260810 18:58:27.235893 28419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:27.243111 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28419
I20260810 18:58:27.243238 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:27.243531 28238 external_mini_cluster.cc:1428] Setting key bd729f2e44a1de03d7c3587503946b1e
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:27.250995 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.237943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:49595 (local address 127.27.147.190:37955)
0810 18:58:27.238093 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:27.238097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:27.239230 (+ 1133us) server_negotiation.cc:407] Connection header received
0810 18:58:27.240182 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:27.240186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:27.240246 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:27.240328 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:27.241555 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.242126 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.243428 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.243779 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.243910 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.246712 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:27.246716 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:27.246719 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:27.246746 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:27.248634 (+ 1888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.249146 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.249152 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.249157 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.249256 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.249680 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.249686 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.249691 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.249955 (+ 264us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:27.250064 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:27.250610 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 18:58:27.250780 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:58:27.251794 28626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.238318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37955 (local address 127.27.147.129:49595)
0810 18:58:27.239061 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:27.239116 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:27.239965 (+ 849us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:27.240515 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:27.240521 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:27.240882 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:27.241398 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:27.241413 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.242263 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.242266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:27.243282 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:27.243297 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.243955 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.243959 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:27.244174 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.244852 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:27.244869 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:27.246545 (+ 1676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:27.248763 (+ 2218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:27.248769 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:27.248782 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:27.249040 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:27.249392 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:27.249395 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:27.249396 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:27.249570 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:27.250076 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:27.250081 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:27.250353 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 18:58:27.250563 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 18:58:27.250921 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":113,"threads_started":1}
I20260810 18:58:27.253159 28623 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37955
I20260810 18:58:27.253460 28623 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:27.254087 28623 heartbeater.cc:511] Master 127.27.147.190:37955 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:27.255811 28327 ts_manager.cc:194] Registered new tserver with Master: 9109d133bc7f4062acd98a409bb5e723 (127.27.147.129:41309)
I20260810 18:58:27.257157 28327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:49595
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:27.274271 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37955
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:27.387966 28630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:27.388187 28630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:27.388228 28630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:27.392139 28630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:27.392194 28630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:27.392278 28630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:27.397593 28630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37955
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:27.398620 28630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:27.399632 28630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:27.409266 28649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:27.409284 28640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:27.409284 28643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:27.409647 28630 server_base.cc:1034] running on GCE node
I20260810 18:58:27.409999 28630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:27.410604 28630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:27.411767 28630 hybrid_clock.cc:648] HybridClock initialized: now 1786388307411739 us; error 35 us; skew 500 ppm
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:27.414408 28630 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:27.415294 28630 webserver.cc:460] Webserver started at http://127.27.147.130:35693/ using document root <none> and password file <none>
I20260810 18:58:27.415841 28630 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:27.415933 28630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:27.416172 28630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:27.418203 28630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "9ed52b359f3a4d018f351ae1d2c8bc4d"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "474fffbcfd6936beae4bcdc56b86226d"
server_key_iv: "d9099e3a765b1b4b6b2f8b134dfa07c9"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.418771 28630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "9ed52b359f3a4d018f351ae1d2c8bc4d"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "474fffbcfd6936beae4bcdc56b86226d"
server_key_iv: "d9099e3a765b1b4b6b2f8b134dfa07c9"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.422597 28630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.001s
I20260810 18:58:27.425241 28662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.426527 28630 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:27.426687 28630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "9ed52b359f3a4d018f351ae1d2c8bc4d"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "474fffbcfd6936beae4bcdc56b86226d"
server_key_iv: "d9099e3a765b1b4b6b2f8b134dfa07c9"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.426810 28630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:27.460811 28630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:27.463742 28630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:27.463948 28630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:27.464679 28630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:27.465678 28630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:27.465750 28630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.465822 28630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:27.465866 28630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.475733 28630 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:34685
I20260810 18:58:27.475762 28846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:34685 every 8 connection(s)
I20260810 18:58:27.476601 28630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:27.481165 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28630
I20260810 18:58:27.481283 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:27.481523 28238 external_mini_cluster.cc:1428] Setting key 6d65d596d7431c948461e7ef41ac0847
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:27.490633 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.478429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:41923 (local address 127.27.147.190:37955)
0810 18:58:27.478568 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:27.478571 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:27.479630 (+ 1059us) server_negotiation.cc:407] Connection header received
0810 18:58:27.480535 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:27.480539 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:27.480597 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:27.480707 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:27.481982 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.482638 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.483875 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.484259 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.484383 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.487126 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:27.487135 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:27.487137 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:27.487159 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:27.488618 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.489141 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.489144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.489146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.489225 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.489563 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.489566 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.489568 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.489750 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:27.489858 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:27.490336 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 18:58:27.490463 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:58:27.491407 28849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.478793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37955 (local address 127.27.147.130:41923)
0810 18:58:27.479484 (+ 691us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:27.479524 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:27.480331 (+ 807us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:27.480877 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:27.480884 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:27.481230 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:27.481811 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:27.481825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.482785 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.482788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:27.483748 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:27.483759 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.484404 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.484407 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:27.484553 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.485189 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:27.485215 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:27.486969 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:27.488735 (+ 1766us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:27.488741 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:27.488754 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:27.489029 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:27.489319 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:27.489323 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:27.489325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:27.489462 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:27.489856 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:27.489863 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:27.490103 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 18:58:27.490277 (+ 174us) client_negotiation.cc:240] Negotiation successful
0810 18:58:27.490577 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:58:27.492707 28847 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37955
I20260810 18:58:27.492956 28847 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:27.493513 28847 heartbeater.cc:511] Master 127.27.147.190:37955 requested a full tablet report, sending...
I20260810 18:58:27.494573 28327 ts_manager.cc:194] Registered new tserver with Master: 9ed52b359f3a4d018f351ae1d2c8bc4d (127.27.147.130:34685)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
I20260810 18:58:27.495435 28327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:41923
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:27.510552 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37955
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:39449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:27.626641 28852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:27.626878 28852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:27.626929 28852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:27.630846 28852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:27.630895 28852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:27.630955 28852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:27.635993 28852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37955
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:27.636955 28852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:27.637853 28852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:27.647301 28872 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:27.647331 28875 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:27.647354 28873 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:27.647488 28852 server_base.cc:1034] running on GCE node
I20260810 18:58:27.648048 28852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:27.648734 28852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:27.649916 28852 hybrid_clock.cc:648] HybridClock initialized: now 1786388307649902 us; error 26 us; skew 500 ppm
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:27.652787 28852 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:27.653680 28852 webserver.cc:460] Webserver started at http://127.27.147.131:46277/ using document root <none> and password file <none>
I20260810 18:58:27.654232 28852 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:27.654325 28852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:27.654568 28852 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:27.656610 28852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "bca789fa9d4b485283097cccfc40e1c4"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "1cdc7b937dd2f746360b765e48c494b2"
server_key_iv: "0318cb3db6c88dc8bfebba093febf0ba"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.657197 28852 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "bca789fa9d4b485283097cccfc40e1c4"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "1cdc7b937dd2f746360b765e48c494b2"
server_key_iv: "0318cb3db6c88dc8bfebba093febf0ba"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.660934 28852 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:58:27.663272 28885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.664440 28852 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:27.664584 28852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "bca789fa9d4b485283097cccfc40e1c4"
format_stamp: "Formatted at 2026-08-10 18:58:27 on dist-test-slave-znh6"
server_key: "1cdc7b937dd2f746360b765e48c494b2"
server_key_iv: "0318cb3db6c88dc8bfebba093febf0ba"
server_key_version: "encryptionkey@0"
I20260810 18:58:27.664696 28852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:27.692176 28852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:27.694938 28852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:27.695173 28852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:27.695770 28852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:27.696676 28852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:27.696751 28852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.696823 28852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:27.696877 28852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:27.706399 28852 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:38741
I20260810 18:58:27.706431 29062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:38741 every 8 connection(s)
I20260810 18:58:27.707281 28852 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:27.707547 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28852
I20260810 18:58:27.707664 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:27.707962 28238 external_mini_cluster.cc:1428] Setting key 36f651b957f8dd6c1c215c7462eebe98
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:27.720126 28382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.710784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:57697 (local address 127.27.147.190:37955)
0810 18:58:27.710954 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:27.710958 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:27.710971 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:58:27.711549 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:27.711552 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:27.711592 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:27.711661 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:27.712724 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.713231 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.714293 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.714503 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.714579 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.716961 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:27.716964 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:27.716966 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:27.716989 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:27.718410 (+ 1421us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.718891 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.718894 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.718895 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.718976 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.719267 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.719268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.719269 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.719432 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:27.719509 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:27.719915 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 18:58:27.720011 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 18:58:27.720829 29070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.709728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37955 (local address 127.27.147.131:57697)
0810 18:58:27.710478 (+ 750us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:27.710529 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:27.711370 (+ 841us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:27.711814 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:27.711820 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:27.712162 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:27.712606 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:27.712619 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.713339 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.713342 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:27.714189 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:27.714198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.714622 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.714625 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:27.714755 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.715276 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:27.715292 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:27.716806 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:27.718514 (+ 1708us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:27.718518 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:27.718528 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:27.718756 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:27.719056 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:27.719059 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:27.719060 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:27.719183 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:27.719520 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:27.719525 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:27.719722 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 18:58:27.719866 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 18:58:27.720079 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":146,"threads_started":1}
I20260810 18:58:27.721949 29064 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37955
I20260810 18:58:27.722186 29064 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:27.722694 29064 heartbeater.cc:511] Master 127.27.147.190:37955 requested a full tablet report, sending...
I20260810 18:58:27.723690 28327 ts_manager.cc:194] Registered new tserver with Master: bca789fa9d4b485283097cccfc40e1c4 (127.27.147.131:38741)
I20260810 18:58:27.724238 28327 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:57697
I20260810 18:58:27.733976 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:27.760875 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.752047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60106 (local address 127.27.147.129:41309)
0810 18:58:27.752370 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:27.752383 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:27.752407 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:58:27.752503 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:27.752508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:27.752686 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:27.752803 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:27.753506 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.754231 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.755129 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.755467 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.755548 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.757754 (+ 2206us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:27.757763 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:27.757774 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:27.757797 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:27.759238 (+ 1441us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.759641 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.759646 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.759647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.759721 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.759965 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.759967 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.759968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.760186 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:27.760327 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:27.760535 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 18:58:27.760694 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
W20260810 18:58:27.761828 28492 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60106
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[28254](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:27.788321 29076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:27.780263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60118 (local address 127.27.147.129:41309)
0810 18:58:27.780418 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:27.780422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:27.780546 (+ 124us) server_negotiation.cc:407] Connection header received
0810 18:58:27.780705 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:27.780708 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:27.780759 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:27.780848 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:27.781508 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.782007 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.782836 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:27.783079 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:27.783193 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:27.785411 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:27.785415 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:27.785417 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:27.785438 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:27.786626 (+ 1188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.787102 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.787106 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.787108 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.787188 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:27.787495 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:27.787499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:27.787501 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:27.787660 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:27.787801 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:27.788036 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 18:58:27.788156 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:58:27.789260 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28419
I20260810 18:58:27.804340 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28630
I20260810 18:58:27.814221 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28852
I20260810 18:58:27.823117 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28274
2026-08-10T18:58:27Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2538 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[29086](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[29086](info): set up 2 sockets
Aug 10 18:58:27 dist-test-slave-znh6 krb5kdc[29086](info): commencing operation
krb5kdc: starting...
W20260810 18:58:29.081111 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.233s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:29 dist-test-slave-znh6 krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:29.202049 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:36023
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:36023
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:29.314558 29104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:29.314798 29104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:29.314908 29104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:29.318717 29104 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:29.318763 29104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:29.318781 29104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:29.318797 29104 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:29.318828 29104 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:29.323906 29104 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:36023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:36023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29104
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:29.324798 29104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:29.325716 29104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:58:29.334275 29104 server_base.cc:1034] running on GCE node
W20260810 18:58:29.334187 29116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:29.334168 29114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:29.334208 29124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:29.335222 29104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:29.336315 29104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:29.337505 29104 hybrid_clock.cc:648] HybridClock initialized: now 1786388309337487 us; error 31 us; skew 500 ppm
Aug 10 18:58:29 dist-test-slave-znh6 krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:29.340282 29104 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:29.341109 29104 webserver.cc:460] Webserver started at http://127.27.147.190:41145/ using document root <none> and password file <none>
I20260810 18:58:29.341624 29104 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:29.341681 29104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:29.341843 29104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:29.343874 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "24a4ded217f04a25bdc6fee4c4bb316f"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "d56cae0bab6ee39026d79d27284ff6c6"
server_key_iv: "f6e1125d6b1142304db11ec924c50137"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.344381 29104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "24a4ded217f04a25bdc6fee4c4bb316f"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "d56cae0bab6ee39026d79d27284ff6c6"
server_key_iv: "f6e1125d6b1142304db11ec924c50137"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.348067 29104 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:58:29.350402 29133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:29.351680 29104 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:29.351781 29104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "24a4ded217f04a25bdc6fee4c4bb316f"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "d56cae0bab6ee39026d79d27284ff6c6"
server_key_iv: "f6e1125d6b1142304db11ec924c50137"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.351862 29104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:29.371011 29104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:29.373751 29104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:29.373915 29104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:29.381116 29104 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:36023
I20260810 18:58:29.381139 29212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:36023 every 8 connection(s)
I20260810 18:58:29.382058 29104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:29.384806 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:29.388173 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29104
I20260810 18:58:29.388262 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:29.388541 28238 external_mini_cluster.cc:1428] Setting key ff4684218144c9ba0cfdb70d0265dcec
I20260810 18:58:29.390918 29213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f: Bootstrap starting.
I20260810 18:58:29.393743 29213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:29.394613 29213 log.cc:824] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:29.396945 29213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f: No bootstrap required, opened a new log
Aug 10 18:58:29 dist-test-slave-znh6 krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:29.401227 29213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36023 } }
I20260810 18:58:29.401537 29213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:29.401600 29213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24a4ded217f04a25bdc6fee4c4bb316f, State: Initialized, Role: FOLLOWER
I20260810 18:58:29.402103 29213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36023 } }
I20260810 18:58:29.402273 29213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:29.402369 29213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:29.402472 29213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:29.403712 29213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36023 } }
I20260810 18:58:29.404115 29213 leader_election.cc:302] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24a4ded217f04a25bdc6fee4c4bb316f; no voters:
I20260810 18:58:29.404467 29213 leader_election.cc:288] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:29.404577 29219 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:29.404842 29219 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [term 1 LEADER]: Becoming Leader. State: Replica: 24a4ded217f04a25bdc6fee4c4bb316f, State: Running, Role: LEADER
I20260810 18:58:29.405381 29219 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36023 } }
I20260810 18:58:29.405570 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:29.389959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.27.147.190:36023)
0810 18:58:29.390386 (+ 427us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:29.390405 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:29.390448 (+ 43us) server_negotiation.cc:407] Connection header received
0810 18:58:29.391299 (+ 851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:29.391325 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:29.391725 (+ 400us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:29.392126 (+ 401us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:29.392785 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.394349 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.395504 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.396178 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.396342 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:29.399240 (+ 2898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:29.399247 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:29.399264 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:29.399295 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:29.402186 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:29.402680 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:29.402689 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:29.402698 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:29.402872 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:29.403141 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:29.403144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:29.403146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:29.403738 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:29.403956 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:29.404267 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 18:58:29.404581 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":113,"threads_started":1}
I20260810 18:58:29.405768 29213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:29.407322 29221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36023 } } }
I20260810 18:58:29.407723 29221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:29.407626 29220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24a4ded217f04a25bdc6fee4c4bb316f. Latest consensus state: current_term: 1 leader_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24a4ded217f04a25bdc6fee4c4bb316f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36023 } } }
I20260810 18:58:29.408766 29220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f [sys.catalog]: This master's current role is: LEADER
W20260810 18:58:29.410724 29239 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:58:29.410809 29239 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:58:29.410939 29240 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:29.411835 29240 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:29.417462 29240 catalog_manager.cc:1347] Generated new cluster ID: 88c01343575646cda3602789c09f1f98
I20260810 18:58:29.417531 29240 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:29.426272 29240 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:29.427366 29240 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:29.443369 29240 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24a4ded217f04a25bdc6fee4c4bb316f: Generated new TSK 0
I20260810 18:58:29.444166 29240 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:29.475749 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:36023
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:29.611414 29255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:29.611685 29255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:29.611764 29255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:29.615792 29255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:29.615876 29255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:29.615962 29255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:29.621030 29255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:36023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29255
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:29.622030 29255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:29.623116 29255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:58:29.633095 29255 server_base.cc:1034] running on GCE node
W20260810 18:58:29.633069 29265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:29.633072 29263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:29.633104 29267 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:29.633710 29255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:29.634406 29255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:29.635598 29255 hybrid_clock.cc:648] HybridClock initialized: now 1786388309635584 us; error 26 us; skew 500 ppm
Aug 10 18:58:29 dist-test-slave-znh6 krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:29.638495 29255 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:29.639467 29255 webserver.cc:460] Webserver started at http://127.27.147.129:43309/ using document root <none> and password file <none>
I20260810 18:58:29.640061 29255 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:29.640157 29255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:29.640406 29255 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:29.642472 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "1eab48839e044e609682ca107f7033f1"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "fd8e3b8e384d964700d166543130d7e4"
server_key_iv: "f1729d9ecb2fc689e4c5922d7ea5324d"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.643095 29255 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "1eab48839e044e609682ca107f7033f1"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "fd8e3b8e384d964700d166543130d7e4"
server_key_iv: "f1729d9ecb2fc689e4c5922d7ea5324d"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.646924 29255 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:58:29.649488 29275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:29.650740 29255 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:29.650914 29255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "1eab48839e044e609682ca107f7033f1"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "fd8e3b8e384d964700d166543130d7e4"
server_key_iv: "f1729d9ecb2fc689e4c5922d7ea5324d"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.651034 29255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:29.663779 29255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:29.667171 29255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:29.667378 29255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:29.667985 29255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:29.668848 29255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:29.668920 29255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:29.668992 29255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:29.669042 29255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:29.678241 29255 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:41419
I20260810 18:58:29.678267 29449 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:41419 every 8 connection(s)
I20260810 18:58:29.679147 29255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:29.682967 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29255
I20260810 18:58:29.683081 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:29.683388 28238 external_mini_cluster.cc:1428] Setting key d7a411a41267bc6d2afb4c7e1b1afdce
Aug 10 18:58:29 dist-test-slave-znh6 krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:29.694339 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:29.680978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:47785 (local address 127.27.147.190:36023)
0810 18:58:29.681134 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:29.681139 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:29.682031 (+ 892us) server_negotiation.cc:407] Connection header received
0810 18:58:29.683598 (+ 1567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:29.683603 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:29.683662 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:29.683761 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:29.685139 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.685978 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.687412 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.687641 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.687725 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:29.690740 (+ 3015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:29.690743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:29.690745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:29.690768 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:29.692296 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:29.692823 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:29.692828 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:29.692830 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:29.692916 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:29.693257 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:29.693262 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:29.693264 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:29.693446 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:29.693557 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:29.694044 (+ 487us) server_negotiation.cc:299] Negotiation successful
0810 18:58:29.694164 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:58:29.695215 29453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:29.681367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:36023 (local address 127.27.147.129:47785)
0810 18:58:29.681854 (+ 487us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:29.681910 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:29.683352 (+ 1442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:29.683966 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:29.683978 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:29.684389 (+ 411us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:29.684986 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:29.685003 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.686105 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.686107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:29.687249 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:29.687263 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.687800 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.687804 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:29.688019 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:29.688643 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:29.688668 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:29.690554 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:29.692428 (+ 1874us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:29.692434 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:29.692448 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:29.692717 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:29.693011 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:29.693015 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:29.693017 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:29.693160 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:29.693556 (+ 396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:29.693562 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:29.693802 (+ 240us) client_negotiation.cc:769] Sending connection context
0810 18:58:29.694003 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 18:58:29.694308 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":114,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 18:58:29.696547 29450 heartbeater.cc:347] Connected to a master server at 127.27.147.190:36023
I20260810 18:58:29.696794 29450 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:29.697404 29450 heartbeater.cc:511] Master 127.27.147.190:36023 requested a full tablet report, sending...
I20260810 18:58:29.699065 29159 ts_manager.cc:194] Registered new tserver with Master: 1eab48839e044e609682ca107f7033f1 (127.27.147.129:41419)
I20260810 18:58:29.700192 29159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:47785
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:29.712028 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:36023
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:29.839897 29456 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:29.840121 29456 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:29.840164 29456 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:29.844128 29456 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:29.844179 29456 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:29.844234 29456 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:29.849293 29456 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:36023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29456
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:29.850277 29456 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:29.851296 29456 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:58:29.861094 29456 server_base.cc:1034] running on GCE node
W20260810 18:58:29.861022 29463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:29.861016 29464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:29.861016 29467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:29.861785 29456 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:29.862375 29456 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:29.863570 29456 hybrid_clock.cc:648] HybridClock initialized: now 1786388309863552 us; error 33 us; skew 500 ppm
Aug 10 18:58:29 dist-test-slave-znh6 krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:29.866235 29456 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:29.867161 29456 webserver.cc:460] Webserver started at http://127.27.147.130:40931/ using document root <none> and password file <none>
I20260810 18:58:29.867715 29456 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:29.867803 29456 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:29.868016 29456 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:29.870028 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "75e8b89c122a47268af7a58841b2d5b4"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "3e948ed17102300b9cfb68bc29454d3e"
server_key_iv: "6ce33e2e1227641c7104636964122bad"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.870687 29456 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "75e8b89c122a47268af7a58841b2d5b4"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "3e948ed17102300b9cfb68bc29454d3e"
server_key_iv: "6ce33e2e1227641c7104636964122bad"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.874239 29456 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:29.876791 29475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:29.877971 29456 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:58:29.878118 29456 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "75e8b89c122a47268af7a58841b2d5b4"
format_stamp: "Formatted at 2026-08-10 18:58:29 on dist-test-slave-znh6"
server_key: "3e948ed17102300b9cfb68bc29454d3e"
server_key_iv: "6ce33e2e1227641c7104636964122bad"
server_key_version: "encryptionkey@0"
I20260810 18:58:29.878223 29456 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:29.892895 29456 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:29.895669 29456 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:29.895856 29456 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:29.896412 29456 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:29.897210 29456 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:29.897294 29456 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:29.897382 29456 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:29.897435 29456 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:29.907006 29456 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:44257
I20260810 18:58:29.907060 29660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:44257 every 8 connection(s)
I20260810 18:58:29.907920 29456 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:29.908250 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29456
I20260810 18:58:29.908368 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:29.908674 28238 external_mini_cluster.cc:1428] Setting key 14bea4fb5b281a21b6d14296036f6714
Aug 10 18:58:29 dist-test-slave-znh6 krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:58:29.922789 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:29.909954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:48677 (local address 127.27.147.190:36023)
0810 18:58:29.910084 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:29.910089 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:29.911234 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 18:58:29.912539 (+ 1305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:29.912543 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:29.912602 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:29.912705 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:29.913826 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.914351 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.915686 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.915913 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.916022 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:29.919048 (+ 3026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:29.919051 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:29.919053 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:29.919079 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:29.920583 (+ 1504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:29.921142 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:29.921144 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:29.921146 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:29.921220 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:29.921592 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:29.921600 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:29.921601 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:29.921784 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:29.921893 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:29.922486 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 18:58:29.922612 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
I20260810 18:58:29.923713 29664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:29.910542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:36023 (local address 127.27.147.130:48677)
0810 18:58:29.911089 (+ 547us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:29.911132 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:29.912332 (+ 1200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:29.912878 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:29.912885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:29.913225 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:29.913680 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:29.913693 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.914498 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.914506 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:29.915546 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:29.915556 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:29.916067 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:29.916070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:29.916242 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:29.916905 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:29.916931 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:29.918865 (+ 1934us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:29.920716 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:29.920721 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:29.920734 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:29.921016 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:29.921320 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:29.921324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:29.921326 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:29.921481 (+ 155us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:29.921892 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:29.921901 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:29.922159 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 18:58:29.922378 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 18:58:29.922718 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":174,"threads_started":1}
I20260810 18:58:29.924993 29662 heartbeater.cc:347] Connected to a master server at 127.27.147.190:36023
I20260810 18:58:29.925299 29662 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:29.925933 29662 heartbeater.cc:511] Master 127.27.147.190:36023 requested a full tablet report, sending...
I20260810 18:58:29.927277 29159 ts_manager.cc:194] Registered new tserver with Master: 75e8b89c122a47268af7a58841b2d5b4 (127.27.147.130:44257)
I20260810 18:58:29.928169 29159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:48677
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:29.941733 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:36023
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:44785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:30.054524 29667 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:30.054759 29667 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:30.054862 29667 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:30.059298 29667 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:30.059351 29667 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:30.059439 29667 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:30.064488 29667 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:44785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:36023
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29667
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:30.065531 29667 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:30.066565 29667 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:30.076368 29677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:30.076382 29680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:30.076378 29676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:30.076547 29667 server_base.cc:1034] running on GCE node
I20260810 18:58:30.077103 29667 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:30.077781 29667 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:30.078986 29667 hybrid_clock.cc:648] HybridClock initialized: now 1786388310078968 us; error 35 us; skew 500 ppm
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:30.081804 29667 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:30.082875 29667 webserver.cc:460] Webserver started at http://127.27.147.131:36607/ using document root <none> and password file <none>
I20260810 18:58:30.083465 29667 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:30.083531 29667 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:30.083757 29667 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:30.085837 29667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "163929108c104cb29c30c79a028fc873"
format_stamp: "Formatted at 2026-08-10 18:58:30 on dist-test-slave-znh6"
server_key: "1c514689455a19383a2a2718a43f4bd4"
server_key_iv: "e1b2e6e4ad096ad8fcefe0ca840520fc"
server_key_version: "encryptionkey@0"
I20260810 18:58:30.086408 29667 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "163929108c104cb29c30c79a028fc873"
format_stamp: "Formatted at 2026-08-10 18:58:30 on dist-test-slave-znh6"
server_key: "1c514689455a19383a2a2718a43f4bd4"
server_key_iv: "e1b2e6e4ad096ad8fcefe0ca840520fc"
server_key_version: "encryptionkey@0"
I20260810 18:58:30.090184 29667 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:58:30.092689 29689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:30.093897 29667 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:58:30.094024 29667 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "163929108c104cb29c30c79a028fc873"
format_stamp: "Formatted at 2026-08-10 18:58:30 on dist-test-slave-znh6"
server_key: "1c514689455a19383a2a2718a43f4bd4"
server_key_iv: "e1b2e6e4ad096ad8fcefe0ca840520fc"
server_key_version: "encryptionkey@0"
I20260810 18:58:30.094126 29667 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:30.137814 29667 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:30.140887 29667 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:30.141101 29667 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:30.141747 29667 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:30.142638 29667 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:30.142714 29667 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:30.142789 29667 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:30.142845 29667 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:30.152355 29667 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:36823
I20260810 18:58:30.152380 29860 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:36823 every 8 connection(s)
I20260810 18:58:30.153257 29667 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:30.159035 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29667
I20260810 18:58:30.159178 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:30.159464 28238 external_mini_cluster.cc:1428] Setting key 367b6ca36f70331210000d328e1561fe
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388310, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:30.167300 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.155198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:54103 (local address 127.27.147.190:36023)
0810 18:58:30.155394 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.155399 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.156237 (+ 838us) server_negotiation.cc:407] Connection header received
0810 18:58:30.157731 (+ 1494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.157734 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.157785 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.157860 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:30.159075 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.159793 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.161299 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.161619 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.161705 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.164165 (+ 2460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:30.164168 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:30.164170 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:30.164193 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:30.165471 (+ 1278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.165931 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.165934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.165935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.166015 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.166306 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.166308 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.166309 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.166475 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:30.166551 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.167081 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.167180 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:58:30.168066 29863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.155601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:36023 (local address 127.27.147.131:54103)
0810 18:58:30.156058 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:30.156137 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:30.157528 (+ 1391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:30.158019 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:30.158026 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:30.158374 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:30.158928 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.158944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.159946 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.159950 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.161176 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.161186 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.161754 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.161757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.161900 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.162514 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:30.162530 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:30.164019 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:30.165575 (+ 1556us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:30.165579 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:30.165590 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:30.165840 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:30.166098 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:30.166100 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:30.166101 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:30.166221 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:30.166564 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:30.166569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:30.166806 (+ 237us) client_negotiation.cc:769] Sending connection context
0810 18:58:30.167023 (+ 217us) client_negotiation.cc:240] Negotiation successful
0810 18:58:30.167253 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":128,"threads_started":1}
I20260810 18:58:30.169173 29861 heartbeater.cc:347] Connected to a master server at 127.27.147.190:36023
I20260810 18:58:30.169397 29861 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:30.169862 29861 heartbeater.cc:511] Master 127.27.147.190:36023 requested a full tablet report, sending...
I20260810 18:58:30.170888 29159 ts_manager.cc:194] Registered new tserver with Master: 163929108c104cb29c30c79a028fc873 (127.27.147.131:36823)
I20260810 18:58:30.171435 29159 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:54103
I20260810 18:58:30.173214 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:58:30.183797 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.176902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47442 (local address 127.27.147.190:36023)
0810 18:58:30.177018 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.177021 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.177210 (+ 189us) server_negotiation.cc:407] Connection header received
0810 18:58:30.177308 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.177310 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.177346 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.177412 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:30.178062 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.178535 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.179396 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.179601 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.179680 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.180980 (+ 1300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:30.180983 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:30.180984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:30.181054 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:30.182338 (+ 1284us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.182706 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.182708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.182709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.182787 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.183078 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.183080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.183081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.183295 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:30.183402 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.183596 (+ 194us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.183682 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:58:30.188318 29159 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:30.191346 29159 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:30.206748 29882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.201297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:41419 (local address 127.0.0.1:56920)
0810 18:58:30.201531 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:30.201542 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:30.201631 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:30.202701 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:30.202705 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:30.202725 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:30.202860 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.202867 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.204198 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.204202 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.205200 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.205210 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.206272 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.206275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.206416 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.206436 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 18:58:30.206478 (+ 42us) client_negotiation.cc:240] Negotiation successful
0810 18:58:30.206572 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":102,"thread_start_us":93,"threads_started":1}
I20260810 18:58:30.207089 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.201144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.27.147.131:36823)
0810 18:58:30.201884 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.201898 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.201920 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:58:30.201995 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.201999 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.202186 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.202361 (+ 175us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:30.202565 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.203771 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.205227 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.205869 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.206543 (+ 674us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.206707 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.206765 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.206899 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":635,"thread_start_us":170,"threads_started":1}
I20260810 18:58:30.207015 29880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.200968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:36823 (local address 127.0.0.1:37112)
0810 18:58:30.201202 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:30.201216 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:30.201324 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:30.202348 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:30.202351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:30.202370 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:30.202457 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.202463 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.203914 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.203918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.205085 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.205094 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.206057 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.206062 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.206194 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.206301 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 18:58:30.206412 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:58:30.206876 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":316,"thread_start_us":91,"threads_started":1}
I20260810 18:58:30.206749 29878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.200250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44257 (local address 127.0.0.1:47766)
0810 18:58:30.200615 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:30.200664 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:30.200794 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:30.201387 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:30.201396 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:30.201428 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:30.201573 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.201583 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.203582 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.203589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.205023 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.205034 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.206207 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.206212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.206331 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.206359 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:58:30.206414 (+ 55us) client_negotiation.cc:240] Negotiation successful
0810 18:58:30.206572 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":71,"thread_start_us":65,"threads_started":1}
I20260810 18:58:30.207580 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.200400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47766 (local address 127.27.147.130:44257)
0810 18:58:30.200796 (+ 396us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.200810 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.200838 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:58:30.200957 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.200962 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.201181 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.202166 (+ 985us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:30.202358 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.203443 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.205157 (+ 1714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.206056 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.207164 (+ 1108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.207298 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.207350 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.207438 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":155,"threads_started":1}
I20260810 18:58:30.207731 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.201815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56920 (local address 127.27.147.129:41419)
0810 18:58:30.202164 (+ 349us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.202170 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.202191 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:58:30.202265 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.202272 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.202450 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.202638 (+ 188us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:30.203004 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.204045 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.205379 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.206118 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.207332 (+ 1214us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.207446 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.207491 (+ 45us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.207573 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":247,"thread_start_us":166,"threads_started":1}
I20260810 18:58:30.210418 29766 tablet_service.cc:1467] Processing CreateTablet for tablet a68ef6ae402f49fc80ec9ce41c88a5cf (DEFAULT_TABLE table=test-table [id=10e558a1938548e19a6888ffbdb765ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:30.210464 29551 tablet_service.cc:1467] Processing CreateTablet for tablet a68ef6ae402f49fc80ec9ce41c88a5cf (DEFAULT_TABLE table=test-table [id=10e558a1938548e19a6888ffbdb765ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:30.210608 29350 tablet_service.cc:1467] Processing CreateTablet for tablet a68ef6ae402f49fc80ec9ce41c88a5cf (DEFAULT_TABLE table=test-table [id=10e558a1938548e19a6888ffbdb765ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:30.211426 29766 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68ef6ae402f49fc80ec9ce41c88a5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:30.211529 29551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68ef6ae402f49fc80ec9ce41c88a5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:30.211529 29350 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a68ef6ae402f49fc80ec9ce41c88a5cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:30.218523 29886 tablet_bootstrap.cc:492] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1: Bootstrap starting.
I20260810 18:58:30.218963 29888 tablet_bootstrap.cc:492] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4: Bootstrap starting.
I20260810 18:58:30.219102 29885 tablet_bootstrap.cc:492] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873: Bootstrap starting.
I20260810 18:58:30.221387 29885 tablet_bootstrap.cc:654] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:30.222062 29888 tablet_bootstrap.cc:654] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:30.222894 29888 log.cc:824] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:30.222894 29885 log.cc:824] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:30.223642 29886 tablet_bootstrap.cc:654] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:30.224431 29886 log.cc:824] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:30.225071 29888 tablet_bootstrap.cc:492] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4: No bootstrap required, opened a new log
I20260810 18:58:30.225071 29885 tablet_bootstrap.cc:492] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873: No bootstrap required, opened a new log
I20260810 18:58:30.225260 29885 ts_tablet_manager.cc:1397] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:58:30.225260 29888 ts_tablet_manager.cc:1397] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:58:30.226202 29886 tablet_bootstrap.cc:492] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1: No bootstrap required, opened a new log
I20260810 18:58:30.226420 29886 ts_tablet_manager.cc:1397] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260810 18:58:30.229220 29885 raft_consensus.cc:348] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.229220 29888 raft_consensus.cc:348] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.229506 29885 raft_consensus.cc:374] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:30.229506 29888 raft_consensus.cc:374] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:30.229568 29888 raft_consensus.cc:729] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75e8b89c122a47268af7a58841b2d5b4, State: Initialized, Role: FOLLOWER
I20260810 18:58:30.229598 29885 raft_consensus.cc:729] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 163929108c104cb29c30c79a028fc873, State: Initialized, Role: FOLLOWER
I20260810 18:58:30.229796 29886 raft_consensus.cc:348] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.230109 29886 raft_consensus.cc:374] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:30.230118 29885 consensus_queue.cc:260] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.230195 29886 raft_consensus.cc:729] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1eab48839e044e609682ca107f7033f1, State: Initialized, Role: FOLLOWER
I20260810 18:58:30.230166 29888 consensus_queue.cc:260] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.230608 29886 consensus_queue.cc:260] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.231160 29888 ts_tablet_manager.cc:1428] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:58:30.231236 29662 heartbeater.cc:503] Master 127.27.147.190:36023 was elected leader, sending a full tablet report...
I20260810 18:58:30.231678 29450 heartbeater.cc:503] Master 127.27.147.190:36023 was elected leader, sending a full tablet report...
I20260810 18:58:30.231976 29886 ts_tablet_manager.cc:1428] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:58:30.232327 29885 ts_tablet_manager.cc:1428] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 18:58:30.232661 29861 heartbeater.cc:503] Master 127.27.147.190:36023 was elected leader, sending a full tablet report...
I20260810 18:58:30.291195 29897 raft_consensus.cc:482] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:30.291467 29897 raft_consensus.cc:504] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.292704 29897 leader_election.cc:288] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 75e8b89c122a47268af7a58841b2d5b4 (127.27.147.130:44257), 163929108c104cb29c30c79a028fc873 (127.27.147.131:36823)
I20260810 18:58:30.296670 29879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.293021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:56595 (local address 127.27.147.130:44257)
0810 18:58:30.293160 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.293164 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.293390 (+ 226us) server_negotiation.cc:407] Connection header received
0810 18:58:30.293575 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.293579 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.293650 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.293699 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:30.294090 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.294595 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.295691 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.296117 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.296251 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.296307 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.296482 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.296552 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:58:30.296705 29898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.292931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44257 (local address 127.27.147.129:56595)
0810 18:58:30.293285 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:30.293303 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:30.293420 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:30.293830 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:30.293832 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:30.293847 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:30.293972 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.293977 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.294712 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.294716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.295560 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.295569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.296240 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.296243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.296357 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.296376 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:58:30.296466 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:58:30.296536 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"spinlock_wait_cycles":896,"thread_start_us":139,"threads_started":1}
I20260810 18:58:30.296975 29881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.293593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:35415 (local address 127.27.147.131:36823)
0810 18:58:30.293726 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.293732 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.293837 (+ 105us) server_negotiation.cc:407] Connection header received
0810 18:58:30.294018 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.294025 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.294067 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.294152 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:30.294423 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.294980 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.296000 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.296420 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.296547 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.296595 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.296785 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.296872 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:58:30.296981 29899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.293399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:36823 (local address 127.27.147.129:35415)
0810 18:58:30.293726 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:30.293740 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:30.293883 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:30.294163 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:30.294165 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:30.294181 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:30.294293 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.294301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.295087 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.295092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.295865 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:30.295873 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.296540 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.296544 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:30.296642 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.296664 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:58:30.296768 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 18:58:30.296835 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":97,"threads_started":1}
I20260810 18:58:30.297596 29590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68ef6ae402f49fc80ec9ce41c88a5cf" candidate_uuid: "1eab48839e044e609682ca107f7033f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e8b89c122a47268af7a58841b2d5b4" is_pre_election: true
I20260810 18:58:30.297780 29794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68ef6ae402f49fc80ec9ce41c88a5cf" candidate_uuid: "1eab48839e044e609682ca107f7033f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163929108c104cb29c30c79a028fc873" is_pre_election: true
I20260810 18:58:30.297988 29590 raft_consensus.cc:2454] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eab48839e044e609682ca107f7033f1 in term 0.
I20260810 18:58:30.298035 29794 raft_consensus.cc:2454] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1eab48839e044e609682ca107f7033f1 in term 0.
I20260810 18:58:30.298513 29277 leader_election.cc:302] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 163929108c104cb29c30c79a028fc873, 1eab48839e044e609682ca107f7033f1; no voters:
I20260810 18:58:30.298768 29897 raft_consensus.cc:2781] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:30.298885 29897 raft_consensus.cc:482] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:30.298945 29897 raft_consensus.cc:3037] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:30.300009 29897 raft_consensus.cc:504] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.300336 29897 leader_election.cc:288] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [CANDIDATE]: Term 1 election: Requested vote from peers 75e8b89c122a47268af7a58841b2d5b4 (127.27.147.130:44257), 163929108c104cb29c30c79a028fc873 (127.27.147.131:36823)
I20260810 18:58:30.300844 29590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68ef6ae402f49fc80ec9ce41c88a5cf" candidate_uuid: "1eab48839e044e609682ca107f7033f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "75e8b89c122a47268af7a58841b2d5b4"
I20260810 18:58:30.300848 29794 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a68ef6ae402f49fc80ec9ce41c88a5cf" candidate_uuid: "1eab48839e044e609682ca107f7033f1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "163929108c104cb29c30c79a028fc873"
I20260810 18:58:30.301005 29590 raft_consensus.cc:3037] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:30.301035 29794 raft_consensus.cc:3037] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:30.302399 29794 raft_consensus.cc:2454] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eab48839e044e609682ca107f7033f1 in term 1.
I20260810 18:58:30.302399 29590 raft_consensus.cc:2454] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1eab48839e044e609682ca107f7033f1 in term 1.
I20260810 18:58:30.302732 29277 leader_election.cc:302] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 163929108c104cb29c30c79a028fc873, 1eab48839e044e609682ca107f7033f1; no voters:
I20260810 18:58:30.302923 29897 raft_consensus.cc:2781] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:30.303144 29897 raft_consensus.cc:686] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [term 1 LEADER]: Becoming Leader. State: Replica: 1eab48839e044e609682ca107f7033f1, State: Running, Role: LEADER
I20260810 18:58:30.303514 29897 consensus_queue.cc:237] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } }
I20260810 18:58:30.306893 29159 catalog_manager.cc:5495] T a68ef6ae402f49fc80ec9ce41c88a5cf P 1eab48839e044e609682ca107f7033f1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1eab48839e044e609682ca107f7033f1 (127.27.147.129). New cstate: current_term: 1 leader_uuid: "1eab48839e044e609682ca107f7033f1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "75e8b89c122a47268af7a58841b2d5b4" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44257 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "163929108c104cb29c30c79a028fc873" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36823 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1eab48839e044e609682ca107f7033f1" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41419 } health_report { overall_health: HEALTHY } } }
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:30.332541 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.323329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56928 (local address 127.27.147.129:41419)
0810 18:58:30.323504 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.323509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.323654 (+ 145us) server_negotiation.cc:407] Connection header received
0810 18:58:30.323819 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.323822 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.323874 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.323964 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:30.324673 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.325204 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.326064 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.326288 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.326395 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.328975 (+ 2580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:30.328996 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:30.329011 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:30.329035 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:30.330697 (+ 1662us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.331203 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.331217 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.331218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.331359 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.331648 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.331650 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.331652 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.331889 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:30.332062 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.332184 (+ 122us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.332305 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388310, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:30.365476 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.353642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56930 (local address 127.27.147.129:41419)
0810 18:58:30.353810 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.353815 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.353953 (+ 138us) server_negotiation.cc:407] Connection header received
0810 18:58:30.354148 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.354152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.354195 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.354281 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:30.355009 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.355536 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.356352 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.356753 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.356867 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.362531 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:30.362534 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:30.362537 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:30.362560 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:30.364067 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.364472 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.364474 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.364475 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.364546 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.364777 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.364779 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.364780 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.364943 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:30.365057 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.365260 (+ 203us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.365350 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29086](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29086](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:30.393033 29883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:30.384284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56942 (local address 127.27.147.129:41419)
0810 18:58:30.384433 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:30.384439 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:30.384599 (+ 160us) server_negotiation.cc:407] Connection header received
0810 18:58:30.384728 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:30.384731 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:30.384780 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:30.384859 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:30.385496 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.386246 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.387113 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:30.387477 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:30.387575 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:30.389981 (+ 2406us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:30.389985 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:30.389988 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:30.390013 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:30.391432 (+ 1419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.391844 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.391849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.391850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.391928 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:30.392190 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:30.392195 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:30.392196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:30.392360 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:30.392539 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:30.392755 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 18:58:30.392867 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:58:30.394369 29319 tablet_service.cc:2908] Rejecting scan request for tablet a68ef6ae402f49fc80ec9ce41c88a5cf: Uninitialized: clean time has not yet been initialized
I20260810 18:58:30.396987 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29255
W20260810 18:58:30.406243 29862 tablet.cc:2369] T a68ef6ae402f49fc80ec9ce41c88a5cf P 163929108c104cb29c30c79a028fc873: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:58:30.409319 29663 tablet.cc:2369] T a68ef6ae402f49fc80ec9ce41c88a5cf P 75e8b89c122a47268af7a58841b2d5b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:58:30.414194 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29456
I20260810 18:58:30.427424 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29667
I20260810 18:58:30.439428 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29104
2026-08-10T18:58:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2618 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29925](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29925](info): set up 2 sockets
Aug 10 18:58:30 dist-test-slave-znh6 krb5kdc[29925](info): commencing operation
krb5kdc: starting...
W20260810 18:58:31.678036 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:31 dist-test-slave-znh6 krb5kdc[29925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:31.805497 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:39845
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:39845
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:31.916891 29943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:31.917121 29943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:31.917160 29943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:31.921092 29943 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:31.921141 29943 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:58:31.921159 29943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:31.921176 29943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:31.921209 29943 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:31.921252 29943 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:31.926329 29943 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:39845
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:39845
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:31.927337 29943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:31.928339 29943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:31.937126 29953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:31.937127 29952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:31.937165 29943 server_base.cc:1034] running on GCE node
W20260810 18:58:31.937114 29955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:31.937904 29943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:31.939011 29943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:31.940217 29943 hybrid_clock.cc:648] HybridClock initialized: now 1786388311940185 us; error 43 us; skew 500 ppm
Aug 10 18:58:31 dist-test-slave-znh6 krb5kdc[29925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:31.943169 29943 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:31.944113 29943 webserver.cc:460] Webserver started at http://127.27.147.190:34491/ using document root <none> and password file <none>
I20260810 18:58:31.944700 29943 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:31.944793 29943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:31.945034 29943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:31.947095 29943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "b28c90b5eec04f83999cef5a7430621a"
format_stamp: "Formatted at 2026-08-10 18:58:31 on dist-test-slave-znh6"
server_key: "636600accccff5d5caac97346e6dcfdd"
server_key_iv: "481ffc02d84b08280c53756864be72af"
server_key_version: "encryptionkey@0"
I20260810 18:58:31.947715 29943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "b28c90b5eec04f83999cef5a7430621a"
format_stamp: "Formatted at 2026-08-10 18:58:31 on dist-test-slave-znh6"
server_key: "636600accccff5d5caac97346e6dcfdd"
server_key_iv: "481ffc02d84b08280c53756864be72af"
server_key_version: "encryptionkey@0"
I20260810 18:58:31.951390 29943 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:58:31.953752 29969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:31.954926 29943 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:31.955070 29943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "b28c90b5eec04f83999cef5a7430621a"
format_stamp: "Formatted at 2026-08-10 18:58:31 on dist-test-slave-znh6"
server_key: "636600accccff5d5caac97346e6dcfdd"
server_key_iv: "481ffc02d84b08280c53756864be72af"
server_key_version: "encryptionkey@0"
I20260810 18:58:31.955181 29943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:31.970360 29943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:31.973585 29943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:31.973802 29943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:31.982441 29943 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:39845
I20260810 18:58:31.982461 30054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:39845 every 8 connection(s)
I20260810 18:58:31.983879 29943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:31.986634 30055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:31.991730 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29943
I20260810 18:58:31.991834 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:31.992084 28238 external_mini_cluster.cc:1428] Setting key 494c2a86e6e5dfffe086bd1e4447e5f7
I20260810 18:58:31.992162 30055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a: Bootstrap starting.
I20260810 18:58:31.994544 30055 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:31.995409 30055 log.cc:824] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:31.997300 30055 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a: No bootstrap required, opened a new log
I20260810 18:58:32.000738 30055 raft_consensus.cc:348] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28c90b5eec04f83999cef5a7430621a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39845 } }
I20260810 18:58:32.001051 30055 raft_consensus.cc:374] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:32.001116 30055 raft_consensus.cc:729] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b28c90b5eec04f83999cef5a7430621a, State: Initialized, Role: FOLLOWER
I20260810 18:58:32.001575 30055 consensus_queue.cc:260] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28c90b5eec04f83999cef5a7430621a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39845 } }
I20260810 18:58:32.001696 30055 raft_consensus.cc:388] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:32.001783 30055 raft_consensus.cc:482] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:32.001915 30055 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 0 FOLLOWER]: Advancing to term 1
Aug 10 18:58:31 dist-test-slave-znh6 krb5kdc[29925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:32.002996 30055 raft_consensus.cc:504] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28c90b5eec04f83999cef5a7430621a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39845 } }
I20260810 18:58:32.003350 30055 leader_election.cc:302] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b28c90b5eec04f83999cef5a7430621a; no voters:
I20260810 18:58:32.003645 30055 leader_election.cc:288] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:32.003744 30065 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:32.003978 30065 raft_consensus.cc:686] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [term 1 LEADER]: Becoming Leader. State: Replica: b28c90b5eec04f83999cef5a7430621a, State: Running, Role: LEADER
I20260810 18:58:32.004340 30065 consensus_queue.cc:237] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28c90b5eec04f83999cef5a7430621a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39845 } }
I20260810 18:58:32.005003 30055 sys_catalog.cc:564] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:32.006064 30066 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b28c90b5eec04f83999cef5a7430621a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28c90b5eec04f83999cef5a7430621a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39845 } } }
I20260810 18:58:32.006183 30066 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:32.006578 30068 sys_catalog.cc:455] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [sys.catalog]: SysCatalogTable state changed. Reason: New leader b28c90b5eec04f83999cef5a7430621a. Latest consensus state: current_term: 1 leader_uuid: "b28c90b5eec04f83999cef5a7430621a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b28c90b5eec04f83999cef5a7430621a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39845 } } }
I20260810 18:58:32.006757 30068 sys_catalog.cc:458] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:32.007369 30078 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:32.008687 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:31.993484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.27.147.190:39845)
0810 18:58:31.993907 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:31.993929 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:31.993963 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:58:31.994581 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:31.994597 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:31.995041 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:31.995323 (+ 282us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:31.996226 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:31.997200 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:31.998367 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:31.999531 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:31.999670 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.002610 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.002616 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.002628 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.002653 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.004800 (+ 2147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.005389 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.005396 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.005401 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.005548 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.005946 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.005952 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.005953 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.006513 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.007316 (+ 803us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.007493 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.007879 (+ 386us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":139,"threads_started":1}
I20260810 18:58:32.010215 30078 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:32.015885 30078 catalog_manager.cc:1347] Generated new cluster ID: 5398fc3516a74f43b78c7981367f122f
I20260810 18:58:32.015970 30078 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:32.032984 30078 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:32.034157 30078 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:32.038378 30078 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a: Generated new TSK 0
I20260810 18:58:32.039316 30078 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:32.071880 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:39845
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:32.186146 30097 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:32.186386 30097 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:32.186430 30097 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:32.190428 30097 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:32.190485 30097 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:32.190522 30097 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:58:32.190604 30097 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:32.195741 30097 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:39845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30097
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:32.196734 30097 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:32.197744 30097 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:32.207413 30103 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:32.207402 30107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:32.207407 30105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:32.207660 30097 server_base.cc:1034] running on GCE node
I20260810 18:58:32.208119 30097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:32.208707 30097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:32.209900 30097 hybrid_clock.cc:648] HybridClock initialized: now 1786388312209879 us; error 31 us; skew 500 ppm
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:32.212594 30097 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:32.213424 30097 webserver.cc:460] Webserver started at http://127.27.147.129:35847/ using document root <none> and password file <none>
I20260810 18:58:32.213979 30097 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:32.214072 30097 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:32.214303 30097 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:32.216317 30097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "944c6a3a96624bcebcfd829a3f5b901f"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "e715b1a47ea7e22ea13715a88b05ef35"
server_key_iv: "298adb0da611ce3ed41c1670173063da"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.216869 30097 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "944c6a3a96624bcebcfd829a3f5b901f"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "e715b1a47ea7e22ea13715a88b05ef35"
server_key_iv: "298adb0da611ce3ed41c1670173063da"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.220494 30097 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 18:58:32.222752 30115 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:32.223933 30097 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:58:32.224077 30097 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "944c6a3a96624bcebcfd829a3f5b901f"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "e715b1a47ea7e22ea13715a88b05ef35"
server_key_iv: "298adb0da611ce3ed41c1670173063da"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.224185 30097 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:32.259451 30097 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:32.262449 30097 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:32.262660 30097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:32.264192 30097 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:32.264277 30097 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:32.264345 30097 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:32.264398 30097 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:32.275882 30097 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:46163
I20260810 18:58:32.275955 30284 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:46163 every 8 connection(s)
I20260810 18:58:32.277181 30097 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:32.278365 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30097
I20260810 18:58:32.278482 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:32.278777 28238 external_mini_cluster.cc:1428] Setting key cd3f9b8e548dc8048b1d3f82a12fc51f
I20260810 18:58:32.278913 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.265152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:55353 (local address 127.27.147.190:39845)
0810 18:58:32.265282 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.265286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.266338 (+ 1052us) server_negotiation.cc:407] Connection header received
0810 18:58:32.267485 (+ 1147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.267509 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.267575 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.267660 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:32.269428 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.270044 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.271645 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.271893 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.272014 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.274677 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.274681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.274684 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.274712 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.276545 (+ 1833us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.277048 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.277052 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.277053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.277159 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.277509 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.277512 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.277514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.277773 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.277877 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.278579 (+ 702us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.278714 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:58:32.279724 30128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.265689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:39845 (local address 127.27.147.129:55353)
0810 18:58:32.266189 (+ 500us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.266230 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.267195 (+ 965us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.268121 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.268131 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.268594 (+ 463us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:32.269275 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.269297 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.270175 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.270178 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.271532 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.271546 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.272064 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.272068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.272238 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.272810 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:32.272836 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:32.274488 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:32.276681 (+ 2193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.276686 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.276696 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.276940 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.277276 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.277278 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.277280 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.277408 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.277940 (+ 532us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:32.277948 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:32.278277 (+ 329us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.278497 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.278773 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":128,"threads_started":1}
I20260810 18:58:32.281092 30287 heartbeater.cc:347] Connected to a master server at 127.27.147.190:39845
I20260810 18:58:32.281261 30287 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:32.281796 30287 heartbeater.cc:511] Master 127.27.147.190:39845 requested a full tablet report, sending...
I20260810 18:58:32.283538 29990 ts_manager.cc:194] Registered new tserver with Master: 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129:46163)
I20260810 18:58:32.285068 29990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:55353
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:32.291110 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.284097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:54097 (local address 127.27.147.190:39845)
0810 18:58:32.284228 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.284235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.284250 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:58:32.284289 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.284294 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.284343 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.284429 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:32.284923 (+ 494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.285770 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.286623 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.286924 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.287036 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.288105 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.288109 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.288111 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.288135 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.289462 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.289895 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.289900 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.289901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.289985 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.290262 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.290267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.290269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.290458 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.290567 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.290844 (+ 277us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.290960 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:58:32.291123 30297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.283718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:39845 (local address 127.27.147.129:54097)
0810 18:58:32.284030 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.284044 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.284125 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.284469 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.284472 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.284672 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:32.284787 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.284793 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.285920 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.285924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.286498 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.286506 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.287038 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.287041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.287135 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.287631 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:32.287648 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:32.287967 (+ 319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:32.289578 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.289582 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.289584 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.289787 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.290079 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.290083 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.290085 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.290171 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.290558 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:32.290561 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:32.290657 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.290766 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.290932 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":2,"thread_start_us":129,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:32.308894 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:39845
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:32.424315 30299 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:32.424585 30299 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:32.424671 30299 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:32.428733 30299 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:32.428812 30299 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:32.428879 30299 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:58:32.428957 30299 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:32.434011 30299 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:39845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30299
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:32.435065 30299 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:32.436065 30299 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:32.446074 30309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:32.446126 30299 server_base.cc:1034] running on GCE node
W20260810 18:58:32.446182 30308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:32.446086 30315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:32.446732 30299 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:32.447397 30299 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:32.448603 30299 hybrid_clock.cc:648] HybridClock initialized: now 1786388312448570 us; error 42 us; skew 500 ppm
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:32.451346 30299 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:32.452210 30299 webserver.cc:460] Webserver started at http://127.27.147.130:46731/ using document root <none> and password file <none>
I20260810 18:58:32.452783 30299 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:32.452883 30299 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:32.453118 30299 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:32.455195 30299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "952941a1054f4a76baa534ce96fc81ea"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "26bf845870be52dfd2a23eda34c5503d"
server_key_iv: "a81554cefae744f01cdb7290f1418980"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.455760 30299 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "952941a1054f4a76baa534ce96fc81ea"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "26bf845870be52dfd2a23eda34c5503d"
server_key_iv: "a81554cefae744f01cdb7290f1418980"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.459610 30299 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:58:32.462143 30323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:32.463423 30299 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:32.463559 30299 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "952941a1054f4a76baa534ce96fc81ea"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "26bf845870be52dfd2a23eda34c5503d"
server_key_iv: "a81554cefae744f01cdb7290f1418980"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.463673 30299 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:32.486650 30299 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:32.489447 30299 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:32.489656 30299 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:32.491217 30299 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:32.491293 30299 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:32.491361 30299 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:32.491400 30299 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:32.502830 30299 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:43377
I20260810 18:58:32.502852 30484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:43377 every 8 connection(s)
I20260810 18:58:32.503755 30299 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:32.505492 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30299
I20260810 18:58:32.505538 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.491905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:49825 (local address 127.27.147.190:39845)
0810 18:58:32.492050 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.492054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.493676 (+ 1622us) server_negotiation.cc:407] Connection header received
0810 18:58:32.494963 (+ 1287us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.494970 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.495045 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.495148 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:32.496333 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.497228 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.498529 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.498852 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.498965 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.501757 (+ 2792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.501762 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.501765 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.501795 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.503395 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.503900 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.503906 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.503909 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.504001 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.504407 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.504411 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.504412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.504644 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.504746 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.505174 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.505353 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:58:32.505612 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:32.505882 28238 external_mini_cluster.cc:1428] Setting key 0c95ae725a9478f5f88814f01eef7a17
I20260810 18:58:32.506410 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.492997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:39845 (local address 127.27.147.130:49825)
0810 18:58:32.493535 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.493576 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.494665 (+ 1089us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.495338 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.495344 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.495769 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:32.496207 (+ 438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.496220 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.497369 (+ 1149us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.497374 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.498381 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.498392 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.499003 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.499008 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.499179 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.499702 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:32.499730 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:32.501591 (+ 1861us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:32.503515 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.503521 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.503535 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.503793 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.504096 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.504100 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.504102 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.504290 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.504753 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:32.504758 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:32.504966 (+ 208us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.505122 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.505489 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"thread_start_us":141,"threads_started":1}
I20260810 18:58:32.507762 30485 heartbeater.cc:347] Connected to a master server at 127.27.147.190:39845
I20260810 18:58:32.507917 30485 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:32.508474 30485 heartbeater.cc:511] Master 127.27.147.190:39845 requested a full tablet report, sending...
I20260810 18:58:32.509590 29990 ts_manager.cc:194] Registered new tserver with Master: 952941a1054f4a76baa534ce96fc81ea (127.27.147.130:43377)
I20260810 18:58:32.510324 29990 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:49825
I20260810 18:58:32.517606 30498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.509978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:39845 (local address 127.27.147.130:57417)
0810 18:58:32.510229 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.510241 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.510311 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.510892 (+ 581us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.510894 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.511078 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:32.511173 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.511179 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.512136 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.512139 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.512678 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.512686 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.513282 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.513284 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.513352 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.513857 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:32.513873 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:32.514165 (+ 292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:32.516103 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.516105 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.516107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.516290 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.516605 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.516607 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.516608 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.516679 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.517217 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:32.517219 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:32.517288 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.517384 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.517483 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":98,"threads_started":1}
I20260810 18:58:32.517764 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.510038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:57417 (local address 127.27.147.190:39845)
0810 18:58:32.510638 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.510642 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.510657 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:58:32.510688 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.510691 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.510730 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.510808 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:32.511291 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.512006 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.512795 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.513165 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.513281 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.514306 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.514312 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.514318 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.514349 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.515999 (+ 1650us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.516396 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.516403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.516407 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.516503 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.516775 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.516781 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.516784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.517113 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.517231 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.517463 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.517595 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524}
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:32.538467 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:39845
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:32.649775 30501 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:32.650003 30501 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:32.650046 30501 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:32.653920 30501 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:32.653968 30501 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:32.654006 30501 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:58:32.654036 30501 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:32.659236 30501 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:39845
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30501
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:32.660174 30501 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:32.661142 30501 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:32.671113 30511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:32.671119 30508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:32.671119 30507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:32.671440 30501 server_base.cc:1034] running on GCE node
I20260810 18:58:32.671954 30501 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:32.672500 30501 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:32.673656 30501 hybrid_clock.cc:648] HybridClock initialized: now 1786388312673644 us; error 25 us; skew 500 ppm
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:32.676424 30501 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:32.677198 30501 webserver.cc:460] Webserver started at http://127.27.147.131:39033/ using document root <none> and password file <none>
I20260810 18:58:32.677686 30501 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:32.677744 30501 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:32.677901 30501 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:32.679855 30501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "6357a22bedb545d5b171aa51c21be6ac"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "3c76bfa525144a68b2367289d628ab3d"
server_key_iv: "cee95f24a698296cf9be746450b420db"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.680362 30501 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "6357a22bedb545d5b171aa51c21be6ac"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "3c76bfa525144a68b2367289d628ab3d"
server_key_iv: "cee95f24a698296cf9be746450b420db"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.683943 30501 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 18:58:32.686297 30519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:32.687577 30501 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:32.687716 30501 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "6357a22bedb545d5b171aa51c21be6ac"
format_stamp: "Formatted at 2026-08-10 18:58:32 on dist-test-slave-znh6"
server_key: "3c76bfa525144a68b2367289d628ab3d"
server_key_iv: "cee95f24a698296cf9be746450b420db"
server_key_version: "encryptionkey@0"
I20260810 18:58:32.687827 30501 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:32.705489 30501 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:32.708796 30501 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:32.708995 30501 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:32.710441 30501 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:32.710523 30501 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:32.710618 30501 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:32.710670 30501 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:32.722318 30501 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:34579
I20260810 18:58:32.722470 30693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:34579 every 8 connection(s)
I20260810 18:58:32.723420 30501 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:32.724790 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30501
I20260810 18:58:32.724911 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:32.725281 28238 external_mini_cluster.cc:1428] Setting key 165c958f0f3e6042981c58a3fc028117
I20260810 18:58:32.725543 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.711529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:48413 (local address 127.27.147.190:39845)
0810 18:58:32.711671 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.711675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.712668 (+ 993us) server_negotiation.cc:407] Connection header received
0810 18:58:32.713918 (+ 1250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.713922 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.713988 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.714098 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:32.715731 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.716685 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.717991 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.718262 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.718393 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.721219 (+ 2826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.721222 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.721224 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.721262 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.723110 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.723710 (+ 600us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.723713 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.723714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.723847 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.724299 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.724303 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.724304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.724590 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.724725 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.725236 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.725372 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:58:32.726402 30534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.712051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:39845 (local address 127.27.147.131:48413)
0810 18:58:32.712520 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.712561 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.713588 (+ 1027us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.714326 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.714336 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.714794 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:32.715557 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.715580 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.716849 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.716854 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.717838 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.717849 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.718451 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.718455 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.718699 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.719334 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:32.719351 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:32.721067 (+ 1716us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:32.723227 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.723232 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.723258 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.723608 (+ 350us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.723984 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.723988 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.723990 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.724170 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.724707 (+ 537us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:32.724712 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:32.724961 (+ 249us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.725142 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.725422 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":126,"threads_started":1}
I20260810 18:58:32.727923 30694 heartbeater.cc:347] Connected to a master server at 127.27.147.190:39845
I20260810 18:58:32.728123 30694 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:32.728605 30694 heartbeater.cc:511] Master 127.27.147.190:39845 requested a full tablet report, sending...
I20260810 18:58:32.729619 29989 ts_manager.cc:194] Registered new tserver with Master: 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579)
I20260810 18:58:32.730429 29989 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:48413
I20260810 18:58:32.737622 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.730354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:41047 (local address 127.27.147.190:39845)
0810 18:58:32.730496 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.730502 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.730721 (+ 219us) server_negotiation.cc:407] Connection header received
0810 18:58:32.730939 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.730941 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.730980 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.731091 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:32.731574 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.732116 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.733058 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.733290 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.733397 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.734446 (+ 1049us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.734449 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.734452 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.734476 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.735860 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.736342 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.736346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.736348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.736440 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.736761 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.736765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.736767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.736948 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.737066 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.737357 (+ 291us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.737475 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:58:32.737635 30702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.730326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:39845 (local address 127.27.147.131:41047)
0810 18:58:32.730624 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.730636 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.730790 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.731113 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.731116 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.731336 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:32.731444 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.731450 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.732276 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.732280 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.732903 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.732911 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.733413 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.733417 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.733558 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.733986 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:32.733998 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:32.734293 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:32.735991 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.735998 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.736001 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.736219 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.736543 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:32.736547 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:32.736549 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:32.736645 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:32.737056 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:32.737060 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:32.737167 (+ 107us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.737295 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.737447 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":139,"threads_started":1}
I20260810 18:58:32.739286 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:58:32 dist-test-slave-znh6 krb5kdc[29925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:32.767480 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.759374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49656 (local address 127.27.147.190:39845)
0810 18:58:32.759494 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.759503 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.759515 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:58:32.759551 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.759554 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.759591 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.759654 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:32.760343 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.760847 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.761693 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.762037 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.762131 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.764491 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:32.764494 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:32.764496 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:32.764519 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:32.765729 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.766121 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.766124 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.766126 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.766198 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:32.766515 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:32.766519 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:32.766520 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:32.766761 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:32.766933 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.767175 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.767295 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:58:32.770371 29989 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:32.773422 29989 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:32.788196 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.782662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40466 (local address 127.27.147.130:43377)
0810 18:58:32.783050 (+ 388us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.783063 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.784083 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 18:58:32.784200 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.784205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.784376 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.784529 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:32.784752 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.785722 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.787006 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.787573 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.787717 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.787846 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.787937 (+ 91us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.788034 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":270,"thread_start_us":98,"threads_started":1}
I20260810 18:58:32.788220 30723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.782517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43377 (local address 127.0.0.1:40466)
0810 18:58:32.783986 (+ 1469us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.784001 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.784093 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.784525 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.784528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.784547 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:32.784641 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.784647 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.785844 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.785847 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.786887 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.786898 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.787708 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.787711 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.787814 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.787838 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.787909 (+ 71us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.788069 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1405,"mutex_wait_us":97,"thread_start_us":91,"threads_started":1}
I20260810 18:58:32.788412 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.782595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56128 (local address 127.27.147.129:46163)
0810 18:58:32.783234 (+ 639us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.783255 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.783279 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:58:32.783364 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.783371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.783580 (+ 209us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.783708 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:32.784039 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.785162 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.786884 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.787486 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.787589 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.787728 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.788149 (+ 421us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.788252 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":535,"thread_start_us":80,"threads_started":1}
I20260810 18:58:32.788635 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.782028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:46163 (local address 127.0.0.1:56128)
0810 18:58:32.782918 (+ 890us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.782932 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.783029 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.783735 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.783745 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.783770 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:32.783912 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.783925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.785281 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.785284 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.786718 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.786726 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.787618 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.787623 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.787743 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.787780 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.787897 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.788061 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":819,"mutex_wait_us":94,"thread_start_us":111,"threads_started":1}
I20260810 18:58:32.791566 30396 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0d639836c64a81be9e868127d42cc5 (DEFAULT_TABLE table=test-table [id=fa4985bf70a8448f86ce567313a23a8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:32.791554 30184 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0d639836c64a81be9e868127d42cc5 (DEFAULT_TABLE table=test-table [id=fa4985bf70a8448f86ce567313a23a8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:32.792708 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0d639836c64a81be9e868127d42cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:32.792708 30396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0d639836c64a81be9e868127d42cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:32.797542 30718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.781821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:34579 (local address 127.0.0.1:43196)
0810 18:58:32.782322 (+ 501us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.782368 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.782496 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.783798 (+ 1302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.783801 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.783819 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:32.783944 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.783950 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.785112 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.785118 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.786714 (+ 1596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.786726 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.789114 (+ 2388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.789118 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.789207 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.789228 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.789298 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.797404 (+ 8106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":158,"thread_start_us":83,"threads_started":1}
I20260810 18:58:32.798130 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.782015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43196 (local address 127.27.147.131:34579)
0810 18:58:32.783300 (+ 1285us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.783313 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.783336 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:58:32.783414 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.783424 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.783624 (+ 200us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.783792 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:32.784092 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.785005 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.787787 (+ 2782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.788588 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.788893 (+ 305us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.789033 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.797864 (+ 8831us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.798000 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1148,"thread_start_us":115,"threads_started":1}
I20260810 18:58:32.800325 30726 tablet_bootstrap.cc:492] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Bootstrap starting.
I20260810 18:58:32.800571 30599 tablet_service.cc:1467] Processing CreateTablet for tablet 3d0d639836c64a81be9e868127d42cc5 (DEFAULT_TABLE table=test-table [id=fa4985bf70a8448f86ce567313a23a8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:32.801590 30728 tablet_bootstrap.cc:492] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Bootstrap starting.
I20260810 18:58:32.801666 30599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3d0d639836c64a81be9e868127d42cc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:32.802686 30726 tablet_bootstrap.cc:654] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:32.803360 30728 tablet_bootstrap.cc:654] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:32.803624 30726 log.cc:824] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:32.804103 30728 log.cc:824] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:32.805851 30728 tablet_bootstrap.cc:492] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: No bootstrap required, opened a new log
I20260810 18:58:32.805928 30726 tablet_bootstrap.cc:492] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: No bootstrap required, opened a new log
I20260810 18:58:32.806074 30728 ts_tablet_manager.cc:1397] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 18:58:32.806164 30726 ts_tablet_manager.cc:1397] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:58:32.809186 30728 raft_consensus.cc:348] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.809463 30728 raft_consensus.cc:374] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:32.809520 30728 raft_consensus.cc:729] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952941a1054f4a76baa534ce96fc81ea, State: Initialized, Role: FOLLOWER
I20260810 18:58:32.810009 30728 consensus_queue.cc:260] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.810428 30734 tablet_bootstrap.cc:492] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Bootstrap starting.
I20260810 18:58:32.810717 30726 raft_consensus.cc:348] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.810993 30726 raft_consensus.cc:374] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:32.811028 30728 ts_tablet_manager.cc:1428] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:58:32.811061 30726 raft_consensus.cc:729] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944c6a3a96624bcebcfd829a3f5b901f, State: Initialized, Role: FOLLOWER
I20260810 18:58:32.811415 30485 heartbeater.cc:503] Master 127.27.147.190:39845 was elected leader, sending a full tablet report...
I20260810 18:58:32.811465 30726 consensus_queue.cc:260] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.812131 30287 heartbeater.cc:503] Master 127.27.147.190:39845 was elected leader, sending a full tablet report...
I20260810 18:58:32.812358 30726 ts_tablet_manager.cc:1428] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:58:32.813691 30734 tablet_bootstrap.cc:654] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:32.814577 30734 log.cc:824] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:32.816581 30734 tablet_bootstrap.cc:492] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: No bootstrap required, opened a new log
I20260810 18:58:32.816774 30734 ts_tablet_manager.cc:1397] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:58:32.819818 30734 raft_consensus.cc:348] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.820024 30734 raft_consensus.cc:374] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:32.820080 30734 raft_consensus.cc:729] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6357a22bedb545d5b171aa51c21be6ac, State: Initialized, Role: FOLLOWER
I20260810 18:58:32.820418 30734 consensus_queue.cc:260] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.821010 30694 heartbeater.cc:503] Master 127.27.147.190:39845 was elected leader, sending a full tablet report...
I20260810 18:58:32.821288 30734 ts_tablet_manager.cc:1428] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 18:58:32.857340 30736 raft_consensus.cc:482] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:32.857594 30736 raft_consensus.cc:504] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.858975 30736 leader_election.cc:288] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579), 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129:46163)
I20260810 18:58:32.863503 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.859177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:34579 (local address 127.27.147.130:59619)
0810 18:58:32.859284 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.859301 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.859441 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.859771 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.859775 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.859800 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:32.859924 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.859932 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.860950 (+ 1018us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.860953 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.862245 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.862257 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.863127 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.863130 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.863232 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.863264 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.863344 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.863401 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 18:58:32.863595 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.859334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:59619 (local address 127.27.147.131:34579)
0810 18:58:32.859496 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.859501 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.859518 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:58:32.859575 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.859578 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.859629 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.859715 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:32.860065 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.860843 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.862383 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.863013 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.863143 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.863195 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.863386 (+ 191us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.863461 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 18:58:32.864168 30724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.859383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:51519 (local address 127.27.147.129:46163)
0810 18:58:32.859507 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.859510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.860038 (+ 528us) server_negotiation.cc:407] Connection header received
0810 18:58:32.860136 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.860140 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.860190 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.860268 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:32.860514 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.861154 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.862880 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.863514 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.863642 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.863695 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.864013 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.864079 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 18:58:32.864194 30741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.859179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:46163 (local address 127.27.147.130:51519)
0810 18:58:32.859926 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:32.859940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:32.860046 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:32.860288 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:32.860291 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:32.860307 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:32.860407 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.860412 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.861263 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.861266 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.862709 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:32.862718 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.863789 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.863792 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:32.863867 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.863884 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 18:58:32.863989 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 18:58:32.864054 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":682,"mutex_wait_us":338,"thread_start_us":164,"threads_started":1}
I20260810 18:58:32.864217 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d639836c64a81be9e868127d42cc5" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6357a22bedb545d5b171aa51c21be6ac" is_pre_election: true
I20260810 18:58:32.864557 30626 raft_consensus.cc:2454] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 0.
I20260810 18:58:32.864797 30216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d639836c64a81be9e868127d42cc5" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944c6a3a96624bcebcfd829a3f5b901f" is_pre_election: true
I20260810 18:58:32.865084 30216 raft_consensus.cc:2454] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 0.
I20260810 18:58:32.865069 30325 leader_election.cc:302] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6357a22bedb545d5b171aa51c21be6ac, 952941a1054f4a76baa534ce96fc81ea; no voters:
I20260810 18:58:32.865382 30736 raft_consensus.cc:2781] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:32.865475 30736 raft_consensus.cc:482] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:32.865546 30736 raft_consensus.cc:3037] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:32.866663 30736 raft_consensus.cc:504] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.867082 30736 leader_election.cc:288] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 election: Requested vote from peers 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579), 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129:46163)
I20260810 18:58:32.867425 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d639836c64a81be9e868127d42cc5" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6357a22bedb545d5b171aa51c21be6ac"
I20260810 18:58:32.867446 30216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3d0d639836c64a81be9e868127d42cc5" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944c6a3a96624bcebcfd829a3f5b901f"
I20260810 18:58:32.867539 30626 raft_consensus.cc:3037] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:32.867607 30216 raft_consensus.cc:3037] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:32.868659 30626 raft_consensus.cc:2454] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 1.
I20260810 18:58:32.868916 30216 raft_consensus.cc:2454] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 1.
I20260810 18:58:32.868960 30325 leader_election.cc:302] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6357a22bedb545d5b171aa51c21be6ac, 952941a1054f4a76baa534ce96fc81ea; no voters:
I20260810 18:58:32.869132 30736 raft_consensus.cc:2781] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:32.869362 30736 raft_consensus.cc:686] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 1 LEADER]: Becoming Leader. State: Replica: 952941a1054f4a76baa534ce96fc81ea, State: Running, Role: LEADER
I20260810 18:58:32.869701 30736 consensus_queue.cc:237] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } }
I20260810 18:58:32.872846 29989 catalog_manager.cc:5495] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 952941a1054f4a76baa534ce96fc81ea (127.27.147.130). New cstate: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } health_report { overall_health: HEALTHY } } }
I20260810 18:58:32.889168 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:32.885890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40480 (local address 127.27.147.130:43377)
0810 18:58:32.886011 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:32.886015 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:32.886027 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:58:32.886191 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:32.886193 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:32.886243 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:32.886346 (+ 103us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:32.886565 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.887119 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.887941 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:32.888145 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:32.888219 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:32.888484 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:32.888850 (+ 366us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:32.888947 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:32.889026 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 18:58:32.889071 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:58:32.894779 30216 raft_consensus.cc:1261] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Refusing update from remote peer 952941a1054f4a76baa534ce96fc81ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:32.895181 30626 raft_consensus.cc:1261] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Refusing update from remote peer 952941a1054f4a76baa534ce96fc81ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:32.895531 30736 consensus_queue.cc:1035] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:32.895792 30742 consensus_queue.cc:1035] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:32.901851 30745 mvcc.cc:205] Tried to move back new op lower bound from 7317046529610211328 to 7317046529516777472. Current Snapshot: MvccSnapshot[applied={T|T < 7317046529610211328}]
I20260810 18:58:32.902014 30750 mvcc.cc:205] Tried to move back new op lower bound from 7317046529610211328 to 7317046529516777472. Current Snapshot: MvccSnapshot[applied={T|T < 7317046529610211328}]
I20260810 18:58:32.903709 30747 mvcc.cc:205] Tried to move back new op lower bound from 7317046529610211328 to 7317046529516777472. Current Snapshot: MvccSnapshot[applied={T|T < 7317046529610211328}]
I20260810 18:58:32.913928 29992 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:32.914160 29992 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:32.916692 29992 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a: Sending DeleteTablet for 3 replicas of tablet 3d0d639836c64a81be9e868127d42cc5
I20260810 18:58:32.917474 30184 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d0d639836c64a81be9e868127d42cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:32 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:56128
I20260810 18:58:32.917567 30599 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d0d639836c64a81be9e868127d42cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:32 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:43196
I20260810 18:58:32.917564 30396 tablet_service.cc:1514] Processing DeleteTablet for tablet 3d0d639836c64a81be9e868127d42cc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:32 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:40466
I20260810 18:58:32.918090 30764 tablet_replica.cc:330] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: stopping tablet replica
I20260810 18:58:32.918387 30764 raft_consensus.cc:2229] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:32.918639 29992 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:58:32.918756 30764 raft_consensus.cc:2258] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:32.919116 30762 tablet_replica.cc:330] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: stopping tablet replica
W20260810 18:58:32.919116 29992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:32.919420 30762 raft_consensus.cc:2229] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:32.919729 30762 raft_consensus.cc:2258] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:32.920081 30765 tablet_replica.cc:330] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: stopping tablet replica
I20260810 18:58:32.920354 30765 raft_consensus.cc:2229] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:32.920631 30765 raft_consensus.cc:2258] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:32.921295 30762 ts_tablet_manager.cc:1905] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:32.921415 30764 ts_tablet_manager.cc:1905] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:32.924049 30184 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb890c862fd441b8e4d243347f975d6 (DEFAULT_TABLE table=test-table [id=4d4467324ca04ee1aa7ed772b66a32db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:32.924250 30764 ts_tablet_manager.cc:1918] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:58:32.924350 30764 log.cc:1196] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/3d0d639836c64a81be9e868127d42cc5
I20260810 18:58:32.924299 30762 ts_tablet_manager.cc:1918] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:58:32.924588 30762 log.cc:1196] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/3d0d639836c64a81be9e868127d42cc5
I20260810 18:58:32.924662 30764 ts_tablet_manager.cc:1939] T 3d0d639836c64a81be9e868127d42cc5 P 952941a1054f4a76baa534ce96fc81ea: Deleting consensus metadata
I20260810 18:58:32.925096 30762 ts_tablet_manager.cc:1939] T 3d0d639836c64a81be9e868127d42cc5 P 944c6a3a96624bcebcfd829a3f5b901f: Deleting consensus metadata
I20260810 18:58:32.925064 30599 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb890c862fd441b8e4d243347f975d6 (DEFAULT_TABLE table=test-table [id=4d4467324ca04ee1aa7ed772b66a32db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:32.925340 30599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb890c862fd441b8e4d243347f975d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:32.926793 30765 ts_tablet_manager.cc:1905] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:32.927538 30734 tablet_bootstrap.cc:492] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Bootstrap starting.
I20260810 18:58:32.927704 29973 catalog_manager.cc:4841] TS 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129:46163): tablet 3d0d639836c64a81be9e868127d42cc5 (table test-table [id=fa4985bf70a8448f86ce567313a23a8a]) successfully deleted
I20260810 18:58:32.928259 30396 tablet_service.cc:1467] Processing CreateTablet for tablet 8bb890c862fd441b8e4d243347f975d6 (DEFAULT_TABLE table=test-table [id=4d4467324ca04ee1aa7ed772b66a32db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:32.928499 30396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb890c862fd441b8e4d243347f975d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:32.928567 30734 tablet_bootstrap.cc:654] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:32.929281 30765 ts_tablet_manager.cc:1918] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:58:32.929409 30765 log.cc:1196] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/3d0d639836c64a81be9e868127d42cc5
I20260810 18:58:32.929724 30765 ts_tablet_manager.cc:1939] T 3d0d639836c64a81be9e868127d42cc5 P 6357a22bedb545d5b171aa51c21be6ac: Deleting consensus metadata
I20260810 18:58:32.930018 30734 tablet_bootstrap.cc:492] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: No bootstrap required, opened a new log
I20260810 18:58:32.930095 30734 ts_tablet_manager.cc:1397] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:32.930549 30734 raft_consensus.cc:348] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:32.930663 30734 raft_consensus.cc:374] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:32.930697 30734 raft_consensus.cc:729] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6357a22bedb545d5b171aa51c21be6ac, State: Initialized, Role: FOLLOWER
I20260810 18:58:32.924330 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bb890c862fd441b8e4d243347f975d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:32.930851 30734 consensus_queue.cc:260] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:32.930918 30728 tablet_bootstrap.cc:492] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Bootstrap starting.
I20260810 18:58:32.931114 30734 ts_tablet_manager.cc:1428] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:32.931885 30728 tablet_bootstrap.cc:654] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:32.932991 30726 tablet_bootstrap.cc:492] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Bootstrap starting.
I20260810 18:58:32.933452 30728 tablet_bootstrap.cc:492] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: No bootstrap required, opened a new log
I20260810 18:58:32.933531 30728 ts_tablet_manager.cc:1397] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:32.933916 30726 tablet_bootstrap.cc:654] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:32.933898 30728 raft_consensus.cc:348] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:32.934016 30728 raft_consensus.cc:374] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:32.934064 30728 raft_consensus.cc:729] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952941a1054f4a76baa534ce96fc81ea, State: Initialized, Role: FOLLOWER
I20260810 18:58:32.934206 30728 consensus_queue.cc:260] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:32.934497 30728 ts_tablet_manager.cc:1428] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:32.934631 29972 catalog_manager.cc:4841] TS 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579): tablet 3d0d639836c64a81be9e868127d42cc5 (table test-table [id=fa4985bf70a8448f86ce567313a23a8a]) successfully deleted
I20260810 18:58:32.934772 29972 catalog_manager.cc:4841] TS 952941a1054f4a76baa534ce96fc81ea (127.27.147.130:43377): tablet 3d0d639836c64a81be9e868127d42cc5 (table test-table [id=fa4985bf70a8448f86ce567313a23a8a]) successfully deleted
I20260810 18:58:32.935233 30726 tablet_bootstrap.cc:492] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: No bootstrap required, opened a new log
I20260810 18:58:32.935307 30726 ts_tablet_manager.cc:1397] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:32.935982 30726 raft_consensus.cc:348] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:32.936097 30726 raft_consensus.cc:374] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:32.936144 30726 raft_consensus.cc:729] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944c6a3a96624bcebcfd829a3f5b901f, State: Initialized, Role: FOLLOWER
I20260810 18:58:32.936281 30726 consensus_queue.cc:260] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:32.936605 30726 ts_tablet_manager.cc:1428] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
W20260810 18:58:32.974426 30695 tablet.cc:2369] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:58:33.004891 30487 tablet.cc:2369] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:58:33.008901 30737 raft_consensus.cc:482] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:33.009140 30737 raft_consensus.cc:504] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.010416 30737 leader_election.cc:288] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 952941a1054f4a76baa534ce96fc81ea (127.27.147.130:43377), 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579)
I20260810 18:58:33.016038 30721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.011314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:50567 (local address 127.27.147.131:34579)
0810 18:58:33.011443 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.011448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.011652 (+ 204us) server_negotiation.cc:407] Connection header received
0810 18:58:33.011838 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.011841 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.011893 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.011972 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:33.012344 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.012833 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.014572 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.015197 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.015291 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.015343 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.015801 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.015894 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:58:33.016077 30769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.010728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43377 (local address 127.27.147.129:51107)
0810 18:58:33.011486 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:33.011503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:33.011626 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:33.011920 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:33.011926 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:33.011950 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:33.012080 (+ 130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.012087 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.013128 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.013131 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.014419 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.014431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.015475 (+ 1044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.015478 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.015632 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.015676 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 18:58:33.015750 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:58:33.015856 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"mutex_wait_us":154,"thread_start_us":112,"threads_started":1}
I20260810 18:58:33.016085 30772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.011145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:34579 (local address 127.27.147.129:50567)
0810 18:58:33.011576 (+ 431us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:33.011588 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:33.011715 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:33.012076 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:33.012079 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:33.012094 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:33.012219 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.012225 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.012945 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.012950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.014419 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.014431 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.015459 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.015463 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.015627 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.015671 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 18:58:33.015750 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 18:58:33.015856 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"mutex_wait_us":96,"thread_start_us":123,"threads_started":1}
I20260810 18:58:33.016606 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.010768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:51107 (local address 127.27.147.130:43377)
0810 18:58:33.010944 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.010950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.011592 (+ 642us) server_negotiation.cc:407] Connection header received
0810 18:58:33.011748 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.011751 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.011814 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.011898 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:33.012198 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.012992 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.014572 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.015197 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.015291 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.015343 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.016407 (+ 1064us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.016488 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 18:58:33.016844 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb890c862fd441b8e4d243347f975d6" candidate_uuid: "944c6a3a96624bcebcfd829a3f5b901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6357a22bedb545d5b171aa51c21be6ac" is_pre_election: true
I20260810 18:58:33.017017 30626 raft_consensus.cc:2454] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944c6a3a96624bcebcfd829a3f5b901f in term 0.
I20260810 18:58:33.017140 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb890c862fd441b8e4d243347f975d6" candidate_uuid: "944c6a3a96624bcebcfd829a3f5b901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952941a1054f4a76baa534ce96fc81ea" is_pre_election: true
I20260810 18:58:33.017364 30421 raft_consensus.cc:2454] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 944c6a3a96624bcebcfd829a3f5b901f in term 0.
I20260810 18:58:33.017445 30118 leader_election.cc:302] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6357a22bedb545d5b171aa51c21be6ac, 944c6a3a96624bcebcfd829a3f5b901f; no voters:
I20260810 18:58:33.017719 30737 raft_consensus.cc:2781] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:33.017800 30737 raft_consensus.cc:482] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:33.017843 30737 raft_consensus.cc:3037] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:33.018944 30737 raft_consensus.cc:504] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.019312 30737 leader_election.cc:288] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [CANDIDATE]: Term 1 election: Requested vote from peers 952941a1054f4a76baa534ce96fc81ea (127.27.147.130:43377), 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579)
I20260810 18:58:33.019670 30421 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb890c862fd441b8e4d243347f975d6" candidate_uuid: "944c6a3a96624bcebcfd829a3f5b901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "952941a1054f4a76baa534ce96fc81ea"
I20260810 18:58:33.019699 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bb890c862fd441b8e4d243347f975d6" candidate_uuid: "944c6a3a96624bcebcfd829a3f5b901f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6357a22bedb545d5b171aa51c21be6ac"
I20260810 18:58:33.019812 30421 raft_consensus.cc:3037] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:33.019832 30626 raft_consensus.cc:3037] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:33.020886 30421 raft_consensus.cc:2454] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944c6a3a96624bcebcfd829a3f5b901f in term 1.
I20260810 18:58:33.021060 30626 raft_consensus.cc:2454] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 944c6a3a96624bcebcfd829a3f5b901f in term 1.
I20260810 18:58:33.021178 30118 leader_election.cc:302] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944c6a3a96624bcebcfd829a3f5b901f, 952941a1054f4a76baa534ce96fc81ea; no voters:
I20260810 18:58:33.021333 30737 raft_consensus.cc:2781] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:33.021595 30737 raft_consensus.cc:686] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 LEADER]: Becoming Leader. State: Replica: 944c6a3a96624bcebcfd829a3f5b901f, State: Running, Role: LEADER
I20260810 18:58:33.022142 30737 consensus_queue.cc:237] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.024787 29992 catalog_manager.cc:5495] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f reported cstate change: term changed from 0 to 1, leader changed from <none> to 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129). New cstate: current_term: 1 leader_uuid: "944c6a3a96624bcebcfd829a3f5b901f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } health_report { overall_health: UNKNOWN } } }
W20260810 18:58:33.028597 30288 tablet.cc:2369] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 18:58:33 dist-test-slave-znh6 krb5kdc[29925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388311, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:33.069836 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.060662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49672 (local address 127.27.147.190:39845)
0810 18:58:33.060796 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.060801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.060979 (+ 178us) server_negotiation.cc:407] Connection header received
0810 18:58:33.061131 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.061134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.061179 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.061273 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:33.061913 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.062608 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.063483 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.063703 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.063809 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.066347 (+ 2538us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:33.066350 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:33.066352 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:33.066376 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:33.068065 (+ 1689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:33.068585 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:33.068588 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:33.068589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:33.068683 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:33.068955 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:33.068965 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:33.068966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:33.069129 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:33.069204 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.069522 (+ 318us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.069683 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":51,"server-negotiator.queue_time_us":45}
I20260810 18:58:33.069869 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.060599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:39845 (local address 127.0.0.1:49672)
0810 18:58:33.060878 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:33.060890 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:33.061025 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:33.061291 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:33.061293 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:33.061703 (+ 410us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:33.061794 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.061801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.062714 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.062721 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.063356 (+ 635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.063365 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.063813 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.063816 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.063894 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.064504 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:33.064516 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:33.066195 (+ 1679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:33.068168 (+ 1973us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:33.068178 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:33.068184 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:33.068476 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:33.068767 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:33.068770 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:33.068772 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:33.068871 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:33.069223 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:33.069229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:33.069356 (+ 127us) client_negotiation.cc:769] Sending connection context
0810 18:58:33.069469 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 18:58:33.069621 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260810 18:58:33.074038 29992 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:49672:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:58:33.074862 29992 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:33.079794 30184 tablet_service.cc:1467] Processing CreateTablet for tablet c3e8560024c54959a7946f67453327a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7108753631f642038e3181bb161ef886]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:58:33.079864 30599 tablet_service.cc:1467] Processing CreateTablet for tablet c3e8560024c54959a7946f67453327a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7108753631f642038e3181bb161ef886]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:58:33.079885 30396 tablet_service.cc:1467] Processing CreateTablet for tablet c3e8560024c54959a7946f67453327a2 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7108753631f642038e3181bb161ef886]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:58:33.080219 30599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e8560024c54959a7946f67453327a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:33.080219 30184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e8560024c54959a7946f67453327a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:33.080295 30396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3e8560024c54959a7946f67453327a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:33.082835 30728 tablet_bootstrap.cc:492] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: Bootstrap starting.
I20260810 18:58:33.083139 30734 tablet_bootstrap.cc:492] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: Bootstrap starting.
I20260810 18:58:33.083487 30726 tablet_bootstrap.cc:492] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: Bootstrap starting.
I20260810 18:58:33.083778 30728 tablet_bootstrap.cc:654] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:33.084146 30734 tablet_bootstrap.cc:654] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:33.084512 30726 tablet_bootstrap.cc:654] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:33.084765 30728 tablet_bootstrap.cc:492] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: No bootstrap required, opened a new log
I20260810 18:58:33.084843 30728 ts_tablet_manager.cc:1397] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:33.085299 30728 raft_consensus.cc:348] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.085425 30728 raft_consensus.cc:374] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:33.085448 30734 tablet_bootstrap.cc:492] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: No bootstrap required, opened a new log
I20260810 18:58:33.085484 30728 raft_consensus.cc:729] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 952941a1054f4a76baa534ce96fc81ea, State: Initialized, Role: FOLLOWER
I20260810 18:58:33.085527 30734 ts_tablet_manager.cc:1397] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:33.085626 30728 consensus_queue.cc:260] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.085919 30728 ts_tablet_manager.cc:1428] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:33.086022 30726 tablet_bootstrap.cc:492] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: No bootstrap required, opened a new log
I20260810 18:58:33.086017 30734 raft_consensus.cc:348] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.086102 30726 ts_tablet_manager.cc:1397] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:33.086135 30734 raft_consensus.cc:374] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:33.086176 30734 raft_consensus.cc:729] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6357a22bedb545d5b171aa51c21be6ac, State: Initialized, Role: FOLLOWER
I20260810 18:58:33.086306 30734 consensus_queue.cc:260] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.086593 30734 ts_tablet_manager.cc:1428] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:33.086539 30726 raft_consensus.cc:348] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.086671 30726 raft_consensus.cc:374] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:33.086711 30726 raft_consensus.cc:729] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 944c6a3a96624bcebcfd829a3f5b901f, State: Initialized, Role: FOLLOWER
I20260810 18:58:33.086669 30740 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.086859 30740 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:33.086859 30726 consensus_queue.cc:260] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.087141 30726 ts_tablet_manager.cc:1428] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:33.087194 30737 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.087301 30737 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:33.087440 30759 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.087560 30759 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:33.332686 30759 raft_consensus.cc:482] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:33.332841 30759 raft_consensus.cc:504] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.333407 30759 leader_election.cc:288] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129:46163), 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579)
I20260810 18:58:33.333882 30216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e8560024c54959a7946f67453327a2" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944c6a3a96624bcebcfd829a3f5b901f" is_pre_election: true
I20260810 18:58:33.333967 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e8560024c54959a7946f67453327a2" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6357a22bedb545d5b171aa51c21be6ac" is_pre_election: true
I20260810 18:58:33.334084 30216 raft_consensus.cc:2454] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 0.
I20260810 18:58:33.334151 30626 raft_consensus.cc:2454] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 0.
I20260810 18:58:33.334462 30327 leader_election.cc:302] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944c6a3a96624bcebcfd829a3f5b901f, 952941a1054f4a76baa534ce96fc81ea; no voters:
I20260810 18:58:33.334636 30759 raft_consensus.cc:2781] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:33.334698 30759 raft_consensus.cc:482] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:33.334744 30759 raft_consensus.cc:3037] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:33.336004 30759 raft_consensus.cc:504] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.336347 30759 leader_election.cc:288] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 election: Requested vote from peers 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129:46163), 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579)
I20260810 18:58:33.336664 30216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e8560024c54959a7946f67453327a2" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "944c6a3a96624bcebcfd829a3f5b901f"
I20260810 18:58:33.336759 30626 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3e8560024c54959a7946f67453327a2" candidate_uuid: "952941a1054f4a76baa534ce96fc81ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6357a22bedb545d5b171aa51c21be6ac"
I20260810 18:58:33.336797 30216 raft_consensus.cc:3037] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:33.336898 30626 raft_consensus.cc:3037] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:33.337836 30216 raft_consensus.cc:2454] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 1.
I20260810 18:58:33.337997 30626 raft_consensus.cc:2454] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 952941a1054f4a76baa534ce96fc81ea in term 1.
I20260810 18:58:33.338167 30327 leader_election.cc:302] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 944c6a3a96624bcebcfd829a3f5b901f, 952941a1054f4a76baa534ce96fc81ea; no voters:
I20260810 18:58:33.338371 30759 raft_consensus.cc:2781] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:33.338496 30759 raft_consensus.cc:686] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [term 1 LEADER]: Becoming Leader. State: Replica: 952941a1054f4a76baa534ce96fc81ea, State: Running, Role: LEADER
I20260810 18:58:33.338642 30759 consensus_queue.cc:237] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } }
I20260810 18:58:33.339416 30736 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: TxnStatusTablet state changed. Reason: New leader 952941a1054f4a76baa534ce96fc81ea. Latest consensus state: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.339533 30736 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:58:33.339861 30789 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:58:33.340131 30789 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:58:33.340576 29990 catalog_manager.cc:5495] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 952941a1054f4a76baa534ce96fc81ea (127.27.147.130). New cstate: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } health_report { overall_health: UNKNOWN } } }
I20260810 18:58:33.382220 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.378584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40486 (local address 127.27.147.130:43377)
0810 18:58:33.378714 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.378718 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.378782 (+ 64us) server_negotiation.cc:407] Connection header received
0810 18:58:33.378991 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.378994 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.379042 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.379111 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:33.379393 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.379900 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.381094 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.381315 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.381399 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.381745 (+ 346us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:33.381839 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:33.381896 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.382028 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.382098 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:58:33.382238 30783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.378473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43377 (local address 127.0.0.1:40486)
0810 18:58:33.378707 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:33.378722 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:33.378862 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:33.379153 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:33.379156 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:33.379164 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:58:33.379290 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.379296 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.380002 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.380006 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.380978 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.380989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.381425 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.381429 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.381520 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.381575 (+ 55us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:58:33.381922 (+ 347us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:58:33.381936 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 18:58:33.382008 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:58:33.382094 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131}
I20260810 18:58:33.385149 30216 raft_consensus.cc:1261] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Refusing update from remote peer 952941a1054f4a76baa534ce96fc81ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:33.385196 30626 raft_consensus.cc:1261] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Refusing update from remote peer 952941a1054f4a76baa534ce96fc81ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:33.385682 30736 consensus_queue.cc:1035] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:33.385825 30759 consensus_queue.cc:1035] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:33.387246 30737 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: TxnStatusTablet state changed. Reason: New leader 952941a1054f4a76baa534ce96fc81ea. Latest consensus state: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.387362 30737 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:33.387573 30740 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: TxnStatusTablet state changed. Reason: New leader 952941a1054f4a76baa534ce96fc81ea. Latest consensus state: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.387688 30740 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:33.389390 30736 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.389495 30736 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:58:33.389609 30736 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.389683 30736 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 952941a1054f4a76baa534ce96fc81ea: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:58:33.390128 30740 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.390291 30740 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 6357a22bedb545d5b171aa51c21be6ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:33.392191 30737 tablet_replica.cc:439] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "952941a1054f4a76baa534ce96fc81ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "944c6a3a96624bcebcfd829a3f5b901f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 46163 } } peers { permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 } } peers { permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 } } }
I20260810 18:58:33.392323 30737 tablet_replica.cc:442] T c3e8560024c54959a7946f67453327a2 P 944c6a3a96624bcebcfd829a3f5b901f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:33.399721 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.395871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56130 (local address 127.27.147.129:46163)
0810 18:58:33.396149 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.396156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.396167 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:58:33.396229 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.396231 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.396280 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.396416 (+ 136us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:33.396662 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.397185 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.398120 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.398403 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.398516 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.398800 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:33.399301 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:33.399443 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.399505 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.399561 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260810 18:58:33.410604 30725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.406993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:39221 (local address 127.27.147.130:43377)
0810 18:58:33.407117 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.407122 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.407263 (+ 141us) server_negotiation.cc:407] Connection header received
0810 18:58:33.407348 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.407352 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.407405 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.407492 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:33.407799 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.408548 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.409477 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.409827 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.409931 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.410198 (+ 267us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:33.410278 (+ 80us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:33.410350 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.410426 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.410477 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:58:33.410606 30802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.406889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43377 (local address 127.27.147.129:39221)
0810 18:58:33.407176 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:33.407189 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:33.407277 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:33.407507 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:33.407509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:33.407515 (+ 6us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:58:33.407666 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.407671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.408674 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.408678 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.409368 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.409378 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.409951 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.409954 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.410027 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.410071 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:58:33.410363 (+ 292us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:58:33.410368 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:58:33.410417 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 18:58:33.410477 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":113,"threads_started":1}
I20260810 18:58:33.418215 30626 raft_consensus.cc:1261] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Refusing update from remote peer 944c6a3a96624bcebcfd829a3f5b901f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:33.418304 30421 raft_consensus.cc:1261] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Refusing update from remote peer 944c6a3a96624bcebcfd829a3f5b901f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:33.418712 30774 consensus_queue.cc:1035] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6357a22bedb545d5b171aa51c21be6ac" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34579 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:33.418905 30737 consensus_queue.cc:1035] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [LEADER]: Connected to new peer: Peer: permanent_uuid: "952941a1054f4a76baa534ce96fc81ea" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:33.443797 30807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.439684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:46163 (local address 127.27.147.130:34493)
0810 18:58:33.440094 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:33.440110 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:33.440261 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:33.440643 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:33.440646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:33.440655 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:58:33.440758 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.440764 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.441703 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.441708 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.442333 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:33.442341 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.442868 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.442872 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:33.442966 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.443045 (+ 79us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:58:33.443458 (+ 413us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:58:33.443463 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:58:33.443546 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 18:58:33.443631 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":112,"threads_started":1}
I20260810 18:58:33.443872 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.439764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:34493 (local address 127.27.147.129:46163)
0810 18:58:33.439900 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.439906 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.440179 (+ 273us) server_negotiation.cc:407] Connection header received
0810 18:58:33.440500 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.440503 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.440543 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.440630 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:33.440870 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.441613 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.442486 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.442716 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.442821 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.443207 (+ 386us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:33.443324 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:33.443377 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.443687 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.443740 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:58:33.472040 29989 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:33.472249 29989 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:33.474784 29989 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b28c90b5eec04f83999cef5a7430621a: Sending DeleteTablet for 3 replicas of tablet 8bb890c862fd441b8e4d243347f975d6
I20260810 18:58:33.475401 30184 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb890c862fd441b8e4d243347f975d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:33 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:56128
I20260810 18:58:33.475868 30599 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb890c862fd441b8e4d243347f975d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:33 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:43196
I20260810 18:58:33.476284 30396 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bb890c862fd441b8e4d243347f975d6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:33 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:40466
I20260810 18:58:33.476413 30811 tablet_replica.cc:330] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: stopping tablet replica
I20260810 18:58:33.476585 30811 raft_consensus.cc:2229] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:33.476961 30811 raft_consensus.cc:2258] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:33.477677 30812 tablet_replica.cc:330] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: stopping tablet replica
I20260810 18:58:33.477818 30812 raft_consensus.cc:2229] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:33.477959 30812 raft_consensus.cc:2258] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:33.478156 30811 ts_tablet_manager.cc:1905] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:33.478199 30813 tablet_replica.cc:330] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: stopping tablet replica
I20260810 18:58:33.478349 30813 raft_consensus.cc:2229] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:33.478482 30813 raft_consensus.cc:2258] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:33.479933 30813 ts_tablet_manager.cc:1905] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:33.481530 30812 ts_tablet_manager.cc:1905] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:33.482167 30811 ts_tablet_manager.cc:1918] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:33.482326 30811 log.cc:1196] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/8bb890c862fd441b8e4d243347f975d6
I20260810 18:58:33.482692 30811 ts_tablet_manager.cc:1939] T 8bb890c862fd441b8e4d243347f975d6 P 944c6a3a96624bcebcfd829a3f5b901f: Deleting consensus metadata
I20260810 18:58:33.483631 29973 catalog_manager.cc:4841] TS 944c6a3a96624bcebcfd829a3f5b901f (127.27.147.129:46163): tablet 8bb890c862fd441b8e4d243347f975d6 (table test-table [id=4d4467324ca04ee1aa7ed772b66a32db]) successfully deleted
I20260810 18:58:33.484571 30812 ts_tablet_manager.cc:1918] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:33.484658 30812 log.cc:1196] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/8bb890c862fd441b8e4d243347f975d6
I20260810 18:58:33.484944 30812 ts_tablet_manager.cc:1939] T 8bb890c862fd441b8e4d243347f975d6 P 6357a22bedb545d5b171aa51c21be6ac: Deleting consensus metadata
I20260810 18:58:33.485666 29972 catalog_manager.cc:4841] TS 6357a22bedb545d5b171aa51c21be6ac (127.27.147.131:34579): tablet 8bb890c862fd441b8e4d243347f975d6 (table test-table [id=4d4467324ca04ee1aa7ed772b66a32db]) successfully deleted
I20260810 18:58:33.488782 30813 ts_tablet_manager.cc:1918] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:33.488862 30813 log.cc:1196] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/8bb890c862fd441b8e4d243347f975d6
I20260810 18:58:33.489130 30813 ts_tablet_manager.cc:1939] T 8bb890c862fd441b8e4d243347f975d6 P 952941a1054f4a76baa534ce96fc81ea: Deleting consensus metadata
I20260810 18:58:33.489756 29972 catalog_manager.cc:4841] TS 952941a1054f4a76baa534ce96fc81ea (127.27.147.130:43377): tablet 8bb890c862fd441b8e4d243347f975d6 (table test-table [id=4d4467324ca04ee1aa7ed772b66a32db]) successfully deleted
Aug 10 18:58:33 dist-test-slave-znh6 krb5kdc[29925](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388312, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:33.499301 30798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.478991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56146 (local address 127.27.147.129:46163)
0810 18:58:33.479110 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.479115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.487138 (+ 8023us) server_negotiation.cc:407] Connection header received
0810 18:58:33.487273 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.487276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.487334 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.487416 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:33.488385 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.489239 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.490587 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.490952 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.491050 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.495173 (+ 4123us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:33.495183 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:33.495202 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:33.495232 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:33.497296 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:33.497808 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:33.497815 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:33.497817 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:33.497909 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:33.498214 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:33.498217 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:33.498219 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:33.498542 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:33.498779 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.498991 (+ 212us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.499122 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 18:58:33.499783 30278 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56146
I20260810 18:58:33.511423 30062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:33.501610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.27.147.190:39845)
0810 18:58:33.501750 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:33.501756 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:33.502220 (+ 464us) server_negotiation.cc:407] Connection header received
0810 18:58:33.502315 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:33.502320 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:33.502387 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:33.502488 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:33.503513 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.504402 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.505748 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:33.506089 (+ 341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:33.506178 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:33.507962 (+ 1784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:33.507966 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:33.507969 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:33.507996 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:33.509544 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:33.510033 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:33.510037 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:33.510039 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:33.510122 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:33.510409 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:33.510413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:33.510414 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:33.510624 (+ 210us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:33.510726 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:33.511133 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 18:58:33.511261 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 18:58:33.512269 29989 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:49674
I20260810 18:58:33.513677 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30097
I20260810 18:58:33.526392 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30299
I20260810 18:58:33.541324 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30501
I20260810 18:58:33.552260 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29943
2026-08-10T18:58:33Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (3113 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:33 dist-test-slave-znh6 krb5kdc[30822](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:33 dist-test-slave-znh6 krb5kdc[30822](info): set up 2 sockets
Aug 10 18:58:33 dist-test-slave-znh6 krb5kdc[30822](info): commencing operation
krb5kdc: starting...
W20260810 18:58:34.794548 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.214s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:34 dist-test-slave-znh6 krb5kdc[30822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:34.917860 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35783
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:35783
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:35.028709 30845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:35.028977 30845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:35.029073 30845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:35.033031 30845 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:35.033106 30845 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:58:35.033138 30845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:35.033179 30845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:35.033215 30845 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:35.033252 30845 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:35.038295 30845 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:35783
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35783
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:35.039355 30845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:35.040357 30845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:35.048846 30854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:35.048861 30851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:35.048938 30845 server_base.cc:1034] running on GCE node
W20260810 18:58:35.048861 30852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:35.049677 30845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:35.050697 30845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:35.051846 30845 hybrid_clock.cc:648] HybridClock initialized: now 1786388315051826 us; error 29 us; skew 500 ppm
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:35.054611 30845 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:35.055481 30845 webserver.cc:460] Webserver started at http://127.27.147.190:36189/ using document root <none> and password file <none>
I20260810 18:58:35.056027 30845 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:35.056118 30845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:35.056340 30845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:35.058832 30845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "9bdf65ef5ffa475badaac9e319e1013a"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "429484ed59f9d1aeb0be70a206132116"
server_key_iv: "363599a8332dc6f9ca7953d013b1a59c"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.059425 30845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "9bdf65ef5ffa475badaac9e319e1013a"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "429484ed59f9d1aeb0be70a206132116"
server_key_iv: "363599a8332dc6f9ca7953d013b1a59c"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.062954 30845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:58:35.065341 30866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:35.066497 30845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:35.066649 30845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "9bdf65ef5ffa475badaac9e319e1013a"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "429484ed59f9d1aeb0be70a206132116"
server_key_iv: "363599a8332dc6f9ca7953d013b1a59c"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.066759 30845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:35.095651 30845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:35.098354 30845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:35.098572 30845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:35.106699 30845 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:35783
I20260810 18:58:35.106715 30951 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:35783 every 8 connection(s)
I20260810 18:58:35.107673 30845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:35.110440 30952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:35.114073 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30845
I20260810 18:58:35.114202 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:35.114477 28238 external_mini_cluster.cc:1428] Setting key 68beaec773d3fb849a945a882c390b3c
I20260810 18:58:35.116426 30952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Bootstrap starting.
I20260810 18:58:35.119015 30952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:35.119814 30952 log.cc:824] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:35.121882 30952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: No bootstrap required, opened a new log
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388314, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:35.126350 30952 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdf65ef5ffa475badaac9e319e1013a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35783 } }
I20260810 18:58:35.126646 30952 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:35.126717 30952 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bdf65ef5ffa475badaac9e319e1013a, State: Initialized, Role: FOLLOWER
I20260810 18:58:35.127228 30952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdf65ef5ffa475badaac9e319e1013a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35783 } }
I20260810 18:58:35.127394 30952 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:35.127480 30952 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:35.127594 30952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:35.128684 30952 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdf65ef5ffa475badaac9e319e1013a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35783 } }
I20260810 18:58:35.129051 30952 leader_election.cc:302] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bdf65ef5ffa475badaac9e319e1013a; no voters:
I20260810 18:58:35.129364 30952 leader_election.cc:288] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:35.129477 30960 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:35.129668 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.115887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48386 (local address 127.27.147.190:35783)
0810 18:58:35.116305 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.116320 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.116355 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:58:35.117012 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.117031 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.117368 (+ 337us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.117638 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.118538 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.119660 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.120729 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.121164 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.121276 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.124076 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.124083 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.124099 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.124130 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.126026 (+ 1896us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.126437 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.126443 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.126449 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.126583 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.126902 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.126905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.126907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.127396 (+ 489us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.128204 (+ 808us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.128371 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.128689 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":136,"threads_started":1}
I20260810 18:58:35.129722 30960 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [term 1 LEADER]: Becoming Leader. State: Replica: 9bdf65ef5ffa475badaac9e319e1013a, State: Running, Role: LEADER
I20260810 18:58:35.130199 30960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdf65ef5ffa475badaac9e319e1013a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35783 } }
I20260810 18:58:35.130601 30952 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:35.131848 30965 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bdf65ef5ffa475badaac9e319e1013a. Latest consensus state: current_term: 1 leader_uuid: "9bdf65ef5ffa475badaac9e319e1013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdf65ef5ffa475badaac9e319e1013a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35783 } } }
I20260810 18:58:35.131984 30965 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:35.132395 30964 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bdf65ef5ffa475badaac9e319e1013a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bdf65ef5ffa475badaac9e319e1013a" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35783 } } }
I20260810 18:58:35.132490 30964 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:35.133368 30973 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:35.136229 30973 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:35.141606 30973 catalog_manager.cc:1347] Generated new cluster ID: 68f198fd71af4d60b78b338d4686a7b3
I20260810 18:58:35.141667 30973 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:35.163187 30973 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:35.164155 30973 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:35.175382 30973 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Generated new TSK 0
I20260810 18:58:35.176105 30973 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:35.212121 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35783
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:35.323556 30989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:35.323829 30989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:35.323906 30989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:35.327827 30989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:35.327903 30989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:35.327975 30989 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:58:35.328017 30989 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:58:35.328087 30989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:35.333446 30989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:35783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:35.334470 30989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:35.335500 30989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:35.345208 30997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:35.345208 31002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:35.345347 30998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:35.345480 30989 server_base.cc:1034] running on GCE node
I20260810 18:58:35.346010 30989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:35.346622 30989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:35.347821 30989 hybrid_clock.cc:648] HybridClock initialized: now 1786388315347811 us; error 53 us; skew 500 ppm
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:35.350564 30989 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:35.351486 30989 webserver.cc:460] Webserver started at http://127.27.147.129:45817/ using document root <none> and password file <none>
I20260810 18:58:35.352054 30989 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:35.352149 30989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:35.352386 30989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:35.354400 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "422918142bbe4ac5871a0f75047f4f4a"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "cb4ca1ab1cc55db6ba7c4c447725ddee"
server_key_iv: "482678b414ab8eb8d9aab4fd55dcc874"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.355023 30989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "422918142bbe4ac5871a0f75047f4f4a"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "cb4ca1ab1cc55db6ba7c4c447725ddee"
server_key_iv: "482678b414ab8eb8d9aab4fd55dcc874"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.358793 30989 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:58:35.361303 31013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:35.362514 30989 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:35.362648 30989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "422918142bbe4ac5871a0f75047f4f4a"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "cb4ca1ab1cc55db6ba7c4c447725ddee"
server_key_iv: "482678b414ab8eb8d9aab4fd55dcc874"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.362758 30989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:35.386673 30989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:35.389639 30989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:35.389845 30989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:35.391424 30989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:35.391505 30989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:35.391623 30989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:35.391661 30989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:35.402779 30989 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:34531
I20260810 18:58:35.402897 31195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:34531 every 8 connection(s)
I20260810 18:58:35.404026 30989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:35.406576 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.392171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:45197 (local address 127.27.147.190:35783)
0810 18:58:35.392299 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.392302 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.394078 (+ 1776us) server_negotiation.cc:407] Connection header received
0810 18:58:35.395424 (+ 1346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.395428 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.395494 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.395576 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.396693 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.397513 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.399388 (+ 1875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.399738 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.399868 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.402922 (+ 3054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.402926 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.402929 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.402964 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.404521 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.405106 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.405112 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.405114 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.405218 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.405614 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.405617 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.405619 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.405805 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.405888 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.406341 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.406456 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:58:35.407342 31028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.393287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35783 (local address 127.27.147.129:45197)
0810 18:58:35.393908 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.393953 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.395200 (+ 1247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.395746 (+ 546us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.395753 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.396077 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:35.396536 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.396549 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.397644 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.397647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.399254 (+ 1607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.399271 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.399906 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.399911 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.400094 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.400667 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:35.400689 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:35.402693 (+ 2004us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:35.404622 (+ 1929us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.404629 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.404643 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.404966 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.405320 (+ 354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.405323 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.405324 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.405450 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.405898 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:35.405903 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:35.406135 (+ 232us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.406276 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.406518 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":151,"threads_started":1}
I20260810 18:58:35.408632 31197 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35783
I20260810 18:58:35.408792 31197 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:35.409072 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30989
I20260810 18:58:35.409196 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:35.409336 31197 heartbeater.cc:511] Master 127.27.147.190:35783 requested a full tablet report, sending...
I20260810 18:58:35.409564 28238 external_mini_cluster.cc:1428] Setting key e1668b8136ef779c9056666e5d0ff7c4
I20260810 18:58:35.410957 30889 ts_manager.cc:194] Registered new tserver with Master: 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531)
I20260810 18:58:35.412269 30889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:45197
I20260810 18:58:35.418663 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.411031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:58757 (local address 127.27.147.190:35783)
0810 18:58:35.411141 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.411145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.411431 (+ 286us) server_negotiation.cc:407] Connection header received
0810 18:58:35.411530 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.411533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.411574 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.411652 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.412211 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.412910 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.414049 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.414351 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.414426 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.415672 (+ 1246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.415675 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.415678 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.415702 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.417091 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.417508 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.417513 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.417515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.417594 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.417869 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.417873 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.417875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.418042 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.418147 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.418397 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.418513 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":48}
I20260810 18:58:35.418671 31206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.411051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35783 (local address 127.27.147.129:58757)
0810 18:58:35.411328 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.411342 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.411444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.411695 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.411701 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.411916 (+ 215us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:35.412082 (+ 166us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.412092 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.413043 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.413047 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.413900 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.413920 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.414576 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.414579 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.414680 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.415207 (+ 527us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:35.415220 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:35.415527 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:35.417202 (+ 1675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.417208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.417210 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.417410 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.417685 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.417690 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.417691 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.417778 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.418137 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:35.418140 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:35.418236 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.418343 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.418480 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:35.437814 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35783
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:35.552213 31209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:35.552445 31209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:35.552490 31209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:35.556507 31209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:35.556561 31209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:35.556599 31209 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:58:35.556613 31209 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:58:35.556643 31209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:35.561659 31209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:35783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31209
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:35.562595 31209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:35.563684 31209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:35.573755 31218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:35.573768 31219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:35.573920 31221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:35.574035 31209 server_base.cc:1034] running on GCE node
I20260810 18:58:35.574472 31209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:35.575131 31209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:35.576355 31209 hybrid_clock.cc:648] HybridClock initialized: now 1786388315576341 us; error 27 us; skew 500 ppm
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:35.579121 31209 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:35.580030 31209 webserver.cc:460] Webserver started at http://127.27.147.130:32797/ using document root <none> and password file <none>
I20260810 18:58:35.580698 31209 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:35.580776 31209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:35.580955 31209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:35.583001 31209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "fb8749d8bda29cff5338b9c7cff6016f"
server_key_iv: "5a2ad77967fa035beb3a11aee731ad46"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.583624 31209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "fb8749d8bda29cff5338b9c7cff6016f"
server_key_iv: "5a2ad77967fa035beb3a11aee731ad46"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.587458 31209 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:58:35.589895 31229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:35.591261 31209 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:35.591382 31209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "fb8749d8bda29cff5338b9c7cff6016f"
server_key_iv: "5a2ad77967fa035beb3a11aee731ad46"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.591468 31209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:35.604054 31209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:35.606969 31209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:35.607121 31209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:35.608547 31209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:35.608626 31209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:35.608717 31209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:35.608769 31209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:35.619940 31209 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:38005
I20260810 18:58:35.620028 31394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:38005 every 8 connection(s)
I20260810 18:58:35.620982 31209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:35.622786 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.609416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:50015 (local address 127.27.147.190:35783)
0810 18:58:35.609571 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.609576 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.610517 (+ 941us) server_negotiation.cc:407] Connection header received
0810 18:58:35.611624 (+ 1107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.611628 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.611688 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.611783 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.613439 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.614260 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.615545 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.615863 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.615982 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.618820 (+ 2838us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.618849 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.618854 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.618888 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.620447 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.621069 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.621074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.621079 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.621162 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.621607 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.621613 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.621618 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.621875 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.621999 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.622485 (+ 486us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.622618 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:58:35.623567 31240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.609875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35783 (local address 127.27.147.130:50015)
0810 18:58:35.610337 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.610388 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.611383 (+ 995us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.612023 (+ 640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.612036 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.612526 (+ 490us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:35.613225 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.613249 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.614390 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.614393 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.615414 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.615425 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.616034 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.616039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.616213 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.616779 (+ 566us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:35.616804 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:35.618629 (+ 1825us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:35.620584 (+ 1955us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.620593 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.620608 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.620941 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.621277 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.621283 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.621288 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.621480 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.621996 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:35.622004 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:35.622239 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.622438 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.622745 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":137,"threads_started":1}
I20260810 18:58:35.624372 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31209
I20260810 18:58:35.624516 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:35.624831 28238 external_mini_cluster.cc:1428] Setting key d1ad63f29788b6d5791293ede5dc2b45
I20260810 18:58:35.624943 31395 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35783
I20260810 18:58:35.625079 31395 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:35.625643 31395 heartbeater.cc:511] Master 127.27.147.190:35783 requested a full tablet report, sending...
I20260810 18:58:35.626680 30890 ts_manager.cc:194] Registered new tserver with Master: d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:35.627372 30890 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:50015
I20260810 18:58:35.636821 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.627662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:42853 (local address 127.27.147.190:35783)
0810 18:58:35.627748 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.627751 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.628963 (+ 1212us) server_negotiation.cc:407] Connection header received
0810 18:58:35.629155 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.629161 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.629214 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.629282 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.629765 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.630497 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.631949 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.632236 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.632396 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.633682 (+ 1286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.633685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.633687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.633708 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.635100 (+ 1392us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.635593 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.635595 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.635597 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.635668 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.635982 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.635984 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.635985 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.636201 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.636280 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.636622 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.636712 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:58:35.637040 31402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.627326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35783 (local address 127.27.147.130:42853)
0810 18:58:35.628867 (+ 1541us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.628887 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.629040 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.629397 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.629400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.629560 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:35.629645 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.629649 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.630663 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.630669 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.631804 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.631821 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.632388 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.632393 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.632525 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.633025 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:35.633047 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:35.633536 (+ 489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:35.635214 (+ 1678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.635219 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.635222 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.635491 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.635778 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.635781 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.635783 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.635878 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.636315 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:35.636321 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:35.636443 (+ 122us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.636569 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.636707 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1417,"thread_start_us":99,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:35.654130 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35783
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34965
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:35.768013 31405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:35.768261 31405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:35.768349 31405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:35.772522 31405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:35.772576 31405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:35.772637 31405 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 18:58:35.772670 31405 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:58:35.772756 31405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:35.777861 31405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34965
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:35783
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:35.778932 31405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:35.779999 31405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:35.789529 31413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:35.789529 31414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:35.789567 31417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:35.789777 31405 server_base.cc:1034] running on GCE node
I20260810 18:58:35.790146 31405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:35.790771 31405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:35.792001 31405 hybrid_clock.cc:648] HybridClock initialized: now 1786388315791966 us; error 45 us; skew 500 ppm
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:35.794803 31405 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:35.795768 31405 webserver.cc:460] Webserver started at http://127.27.147.131:46821/ using document root <none> and password file <none>
I20260810 18:58:35.796307 31405 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:35.796370 31405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:35.796578 31405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:35.798527 31405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "671621869510425fb0adefb2d0a8ed3c"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "6272eb32ea430ab7cfd2dc6af2819742"
server_key_iv: "68fb797d4a803b66982c11a9b176109b"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.799115 31405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "671621869510425fb0adefb2d0a8ed3c"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "6272eb32ea430ab7cfd2dc6af2819742"
server_key_iv: "68fb797d4a803b66982c11a9b176109b"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.802625 31405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.003s
I20260810 18:58:35.805002 31429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:35.806152 31405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:58:35.806284 31405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "671621869510425fb0adefb2d0a8ed3c"
format_stamp: "Formatted at 2026-08-10 18:58:35 on dist-test-slave-znh6"
server_key: "6272eb32ea430ab7cfd2dc6af2819742"
server_key_iv: "68fb797d4a803b66982c11a9b176109b"
server_key_version: "encryptionkey@0"
I20260810 18:58:35.806391 31405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:35.837139 31405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:35.839963 31405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:35.840168 31405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:35.841593 31405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:35.841665 31405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:35.841729 31405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:35.841780 31405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:35.853839 31405 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:41613
I20260810 18:58:35.853922 31596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:41613 every 8 connection(s)
I20260810 18:58:35.854993 31405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:35.856796 31444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.843105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35783 (local address 127.27.147.131:46401)
0810 18:58:35.843603 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.843645 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.844638 (+ 993us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.845307 (+ 669us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.845315 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.845734 (+ 419us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:35.846479 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.846505 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.847581 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.847584 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.848468 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.848478 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.849012 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.849016 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.849186 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.849788 (+ 602us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:35.849811 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:35.851789 (+ 1978us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:35.853882 (+ 2093us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.853890 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.853904 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.854264 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.854580 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.854587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.854591 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.854755 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.855277 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:35.855285 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:35.855622 (+ 337us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.855826 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.856117 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":160,"threads_started":1}
I20260810 18:58:35.857435 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.842533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:46401 (local address 127.27.147.190:35783)
0810 18:58:35.842670 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.842674 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.843765 (+ 1091us) server_negotiation.cc:407] Connection header received
0810 18:58:35.844917 (+ 1152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.844922 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.845006 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.845110 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.846633 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.847451 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.848604 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.848841 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.848959 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.852009 (+ 3050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.852012 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.852015 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.852063 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.853760 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.854386 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.854389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.854390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.854475 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.854903 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.854907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.854908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.855171 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.855254 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.857208 (+ 1954us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.857330 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:58:35.858361 31597 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35783
I20260810 18:58:35.858502 31597 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:35.859086 31597 heartbeater.cc:511] Master 127.27.147.190:35783 requested a full tablet report, sending...
I20260810 18:58:35.860157 30889 ts_manager.cc:194] Registered new tserver with Master: 671621869510425fb0adefb2d0a8ed3c (127.27.147.131:41613)
I20260810 18:58:35.860865 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31405
I20260810 18:58:35.860905 30889 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:46401
I20260810 18:58:35.861028 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:35.861330 28238 external_mini_cluster.cc:1428] Setting key 4858c118c069209de5f8f640d8abbd68
I20260810 18:58:35.863675 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:58:35.869668 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.860753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:50907 (local address 127.27.147.190:35783)
0810 18:58:35.860867 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.860872 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.861879 (+ 1007us) server_negotiation.cc:407] Connection header received
0810 18:58:35.861990 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.861995 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.862042 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.862137 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.862722 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.863475 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.864621 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.864824 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.864900 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.866282 (+ 1382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.866286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.866291 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.866323 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.867995 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.868489 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.868493 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.868495 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.868578 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.868859 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.868864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.868866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.869040 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.869148 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.869408 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.869525 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:58:35.869681 31603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.860640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35783 (local address 127.27.147.131:50907)
0810 18:58:35.861800 (+ 1160us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.861816 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.861914 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.862259 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.862262 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.862474 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:35.862610 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.862616 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.863620 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.863623 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.864499 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.864512 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.864942 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.864948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.865082 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.865625 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:35.865644 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:35.866106 (+ 462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:35.868128 (+ 2022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.868131 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.868135 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.868370 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.868673 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:35.868676 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:35.868678 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:35.868766 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:35.869141 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:35.869144 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:35.869243 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.869351 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.869487 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1073,"thread_start_us":79,"threads_started":1}
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:58:35 dist-test-slave-znh6 krb5kdc[30822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:35.892561 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.884254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56614 (local address 127.27.147.190:35783)
0810 18:58:35.884378 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.884383 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.884404 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:58:35.884438 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.884441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.884476 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.884541 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:35.885219 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.885729 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.886562 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.886771 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.886955 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.889594 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:35.889598 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:35.889600 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:35.889623 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:35.890950 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.891481 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.891484 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.891485 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.891560 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:35.891792 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:35.891794 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:35.891795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:35.892062 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:35.892163 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.892356 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.892446 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:58:35.895900 30889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:35.898779 30889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:35.913441 31623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.906986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:41613 (local address 127.0.0.1:42804)
0810 18:58:35.908141 (+ 1155us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.908189 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.908322 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.908918 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.908926 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.908956 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:35.909108 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.909114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.910526 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.910530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.911933 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.911944 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.912914 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.912918 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.913043 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.913061 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.913130 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.913217 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":91,"thread_start_us":78,"threads_started":1}
I20260810 18:58:35.913466 31629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.907988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45700 (local address 127.27.147.129:34531)
0810 18:58:35.908873 (+ 885us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.908879 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.908899 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:58:35.909029 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.909036 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.909218 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.909396 (+ 178us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:35.909659 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.910804 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.912174 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.912691 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.912931 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.913066 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.913195 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.913285 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":75,"threads_started":1}
I20260810 18:58:35.913641 31628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.907049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:38005 (local address 127.0.0.1:37948)
0810 18:58:35.908142 (+ 1093us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.908234 (+ 92us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.908336 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.908987 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.908989 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.909001 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:35.909108 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.909114 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.910447 (+ 1333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.910450 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.911706 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.911716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.912806 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.912812 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.913347 (+ 535us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.913368 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.913436 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.913518 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":997,"mutex_wait_us":574,"thread_start_us":108,"threads_started":1}
I20260810 18:58:35.913846 31622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.907715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:34531 (local address 127.0.0.1:45700)
0810 18:58:35.908565 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:35.908577 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:35.908653 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:35.909389 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:35.909392 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:35.909409 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:35.909537 (+ 128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.909544 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.910951 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.910955 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.912064 (+ 1109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:35.912072 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.912835 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.912838 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:35.913018 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.913058 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 18:58:35.913168 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 18:58:35.913241 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"mutex_wait_us":71,"thread_start_us":94,"threads_started":1}
I20260810 18:58:35.914050 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.907099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42804 (local address 127.27.147.131:41613)
0810 18:58:35.907627 (+ 528us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.907642 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.908275 (+ 633us) server_negotiation.cc:407] Connection header received
0810 18:58:35.908516 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.908522 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.908748 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.908900 (+ 152us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:35.909227 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.910386 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.912062 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.912779 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.913410 (+ 631us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.913556 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.913612 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.913909 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":157,"threads_started":1}
I20260810 18:58:35.914362 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:35.907502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37948 (local address 127.27.147.130:38005)
0810 18:58:35.908581 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:35.908606 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:35.908624 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:58:35.908683 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:35.908686 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:35.908820 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:35.909311 (+ 491us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:35.909476 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.910285 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.911840 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:35.912658 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:35.912829 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:35.912988 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:35.914125 (+ 1137us) server_negotiation.cc:299] Negotiation successful
0810 18:58:35.914224 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":989,"thread_start_us":124,"threads_started":1}
I20260810 18:58:35.916656 31093 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6a57e3d9614f7e972fdc389b33a94d (DEFAULT_TABLE table=test-table [id=db297cfe3b1541cd9b5b8ca230f8cbe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:35.916833 31499 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6a57e3d9614f7e972fdc389b33a94d (DEFAULT_TABLE table=test-table [id=db297cfe3b1541cd9b5b8ca230f8cbe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:35.916837 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 8d6a57e3d9614f7e972fdc389b33a94d (DEFAULT_TABLE table=test-table [id=db297cfe3b1541cd9b5b8ca230f8cbe2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:35.917737 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6a57e3d9614f7e972fdc389b33a94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:35.917915 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6a57e3d9614f7e972fdc389b33a94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:35.917909 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d6a57e3d9614f7e972fdc389b33a94d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:35.924088 31630 tablet_bootstrap.cc:492] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Bootstrap starting.
I20260810 18:58:35.925791 31631 tablet_bootstrap.cc:492] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Bootstrap starting.
I20260810 18:58:35.926447 31630 tablet_bootstrap.cc:654] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:35.926748 31633 tablet_bootstrap.cc:492] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Bootstrap starting.
I20260810 18:58:35.927416 31630 log.cc:824] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:35.928014 31631 tablet_bootstrap.cc:654] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:35.928722 31633 tablet_bootstrap.cc:654] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:35.928859 31631 log.cc:824] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:35.929520 31630 tablet_bootstrap.cc:492] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: No bootstrap required, opened a new log
I20260810 18:58:35.929565 31633 log.cc:824] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:35.929761 31630 ts_tablet_manager.cc:1397] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:58:35.930784 31631 tablet_bootstrap.cc:492] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: No bootstrap required, opened a new log
I20260810 18:58:35.930995 31631 ts_tablet_manager.cc:1397] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 18:58:35.931445 31633 tablet_bootstrap.cc:492] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: No bootstrap required, opened a new log
I20260810 18:58:35.931643 31633 ts_tablet_manager.cc:1397] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20260810 18:58:35.934156 31630 raft_consensus.cc:348] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:35.934408 31630 raft_consensus.cc:374] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:35.934459 31630 raft_consensus.cc:729] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422918142bbe4ac5871a0f75047f4f4a, State: Initialized, Role: FOLLOWER
I20260810 18:58:35.934487 31633 raft_consensus.cc:348] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:35.934698 31633 raft_consensus.cc:374] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:35.934753 31633 raft_consensus.cc:729] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b7ce52d1ca44e1ae52b84f44ca2ac9, State: Initialized, Role: FOLLOWER
I20260810 18:58:35.934736 31631 raft_consensus.cc:348] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:35.934916 31630 consensus_queue.cc:260] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:35.935025 31631 raft_consensus.cc:374] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:35.935078 31631 raft_consensus.cc:729] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Initialized, Role: FOLLOWER
I20260810 18:58:35.935237 31633 consensus_queue.cc:260] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:35.935483 31631 consensus_queue.cc:260] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:35.935647 31197 heartbeater.cc:503] Master 127.27.147.190:35783 was elected leader, sending a full tablet report...
I20260810 18:58:35.936029 31630 ts_tablet_manager.cc:1428] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:58:35.936558 31597 heartbeater.cc:503] Master 127.27.147.190:35783 was elected leader, sending a full tablet report...
I20260810 18:58:35.936609 31631 ts_tablet_manager.cc:1428] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:58:35.937155 31633 ts_tablet_manager.cc:1428] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent starting tablet: real 0.005s user 0.001s sys 0.003s
I20260810 18:58:35.937459 31395 heartbeater.cc:503] Master 127.27.147.190:35783 was elected leader, sending a full tablet report...
W20260810 18:58:36.106091 31598 tablet.cc:2369] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:58:36.122212 31396 tablet.cc:2369] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:58:36.155640 31198 tablet.cc:2369] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:58:36.185017 31641 raft_consensus.cc:482] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:36.185236 31641 raft_consensus.cc:504] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:36.186468 31641 leader_election.cc:288] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 671621869510425fb0adefb2d0a8ed3c (127.27.147.131:41613), 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531)
I20260810 18:58:36.190507 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.186866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:38093 (local address 127.27.147.131:41613)
0810 18:58:36.187006 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.187009 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.187076 (+ 67us) server_negotiation.cc:407] Connection header received
0810 18:58:36.187317 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.187321 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.187391 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.187441 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:36.187832 (+ 391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.188367 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.189486 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.189926 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.190065 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.190125 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.190302 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.190379 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:58:36.190510 31644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.186684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:41613 (local address 127.27.147.130:38093)
0810 18:58:36.186978 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.186996 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.187139 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.187578 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.187580 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.187598 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:36.187704 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.187708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.188498 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.188501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.189345 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.189354 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.190057 (+ 703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.190060 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.190161 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.190190 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.190284 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.190365 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":146,"threads_started":1}
I20260810 18:58:36.190668 31645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.187158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:34531 (local address 127.27.147.130:42575)
0810 18:58:36.187423 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.187438 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.187557 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.187852 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.187855 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.187873 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:36.188000 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.188005 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.188811 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.188815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.189544 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.189552 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.190210 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.190214 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.190321 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.190343 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.190448 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.190519 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":67,"threads_started":1}
I20260810 18:58:36.190655 31629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.187281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:42575 (local address 127.27.147.129:34531)
0810 18:58:36.187420 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.187425 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.187504 (+ 79us) server_negotiation.cc:407] Connection header received
0810 18:58:36.187690 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.187693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.187746 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.187834 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:36.188116 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.188650 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.189677 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.190081 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.190210 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.190261 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.190471 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.190549 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 18:58:36.191356 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6a57e3d9614f7e972fdc389b33a94d" candidate_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671621869510425fb0adefb2d0a8ed3c" is_pre_election: true
I20260810 18:58:36.191362 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6a57e3d9614f7e972fdc389b33a94d" candidate_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a" is_pre_election: true
I20260810 18:58:36.191668 31124 raft_consensus.cc:2454] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b7ce52d1ca44e1ae52b84f44ca2ac9 in term 0.
I20260810 18:58:36.191675 31534 raft_consensus.cc:2454] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8b7ce52d1ca44e1ae52b84f44ca2ac9 in term 0.
I20260810 18:58:36.192183 31235 leader_election.cc:302] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671621869510425fb0adefb2d0a8ed3c, d8b7ce52d1ca44e1ae52b84f44ca2ac9; no voters:
I20260810 18:58:36.192488 31641 raft_consensus.cc:2781] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:36.192569 31641 raft_consensus.cc:482] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:36.192600 31641 raft_consensus.cc:3037] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.193701 31641 raft_consensus.cc:504] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:36.194057 31641 leader_election.cc:288] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [CANDIDATE]: Term 1 election: Requested vote from peers 671621869510425fb0adefb2d0a8ed3c (127.27.147.131:41613), 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531)
I20260810 18:58:36.194435 31534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6a57e3d9614f7e972fdc389b33a94d" candidate_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "671621869510425fb0adefb2d0a8ed3c"
I20260810 18:58:36.194561 31534 raft_consensus.cc:3037] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.194571 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d6a57e3d9614f7e972fdc389b33a94d" candidate_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a"
I20260810 18:58:36.194711 31124 raft_consensus.cc:3037] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.196041 31124 raft_consensus.cc:2454] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b7ce52d1ca44e1ae52b84f44ca2ac9 in term 1.
I20260810 18:58:36.196308 31534 raft_consensus.cc:2454] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8b7ce52d1ca44e1ae52b84f44ca2ac9 in term 1.
I20260810 18:58:36.196367 31235 leader_election.cc:302] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422918142bbe4ac5871a0f75047f4f4a, d8b7ce52d1ca44e1ae52b84f44ca2ac9; no voters:
I20260810 18:58:36.196552 31641 raft_consensus.cc:2781] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:36.196774 31641 raft_consensus.cc:686] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 LEADER]: Becoming Leader. State: Replica: d8b7ce52d1ca44e1ae52b84f44ca2ac9, State: Running, Role: LEADER
I20260810 18:58:36.197131 31641 consensus_queue.cc:237] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } }
I20260810 18:58:36.200287 30889 catalog_manager.cc:5495] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130). New cstate: current_term: 1 leader_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } health_report { overall_health: UNKNOWN } } }
Aug 10 18:58:36 dist-test-slave-znh6 krb5kdc[30822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388315, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:36.301414 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.292610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35783 (local address 127.0.0.1:56618)
0810 18:58:36.292957 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.292981 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.293092 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.293335 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.293337 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.293733 (+ 396us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:36.293847 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.293854 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.294573 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.294575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.295300 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.295310 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.295717 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.295719 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.295800 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.296440 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:36.296452 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:36.297927 (+ 1475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:36.299812 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:36.299823 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:36.299829 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:36.300068 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:36.300352 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:36.300356 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:36.300358 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:36.300441 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:36.300815 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:36.300823 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:36.300938 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.301055 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.301208 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":134,"threads_started":1}
I20260810 18:58:36.301419 30958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.292653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56618 (local address 127.27.147.190:35783)
0810 18:58:36.292803 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.292809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.293068 (+ 259us) server_negotiation.cc:407] Connection header received
0810 18:58:36.293176 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.293179 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.293221 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.293291 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:36.293966 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.294480 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.295404 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.295615 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.295694 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.298078 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:36.298081 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:36.298083 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:36.298105 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:36.299686 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:36.300179 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:36.300182 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:36.300184 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:36.300262 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:36.300531 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:36.300536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:36.300538 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:36.300708 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:36.300815 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.301113 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.301263 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":57}
I20260810 18:58:36.305348 30889 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:56618:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:58:36.306017 30889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:36.311160 31499 tablet_service.cc:1467] Processing CreateTablet for tablet 24140099e5a94465a01eb912ba846237 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64abb7275e7c42d584a77ca30daa0210]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:58:36.311137 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 24140099e5a94465a01eb912ba846237 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64abb7275e7c42d584a77ca30daa0210]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:58:36.311249 31093 tablet_service.cc:1467] Processing CreateTablet for tablet 24140099e5a94465a01eb912ba846237 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=64abb7275e7c42d584a77ca30daa0210]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:58:36.311885 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24140099e5a94465a01eb912ba846237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:36.311892 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24140099e5a94465a01eb912ba846237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:36.311951 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 24140099e5a94465a01eb912ba846237. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:36.314620 31630 tablet_bootstrap.cc:492] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: Bootstrap starting.
I20260810 18:58:36.314741 31633 tablet_bootstrap.cc:492] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Bootstrap starting.
I20260810 18:58:36.314931 31631 tablet_bootstrap.cc:492] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: Bootstrap starting.
I20260810 18:58:36.315728 31630 tablet_bootstrap.cc:654] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:36.315908 31631 tablet_bootstrap.cc:654] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:36.316193 31633 tablet_bootstrap.cc:654] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:36.317338 31631 tablet_bootstrap.cc:492] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: No bootstrap required, opened a new log
I20260810 18:58:36.317438 31631 ts_tablet_manager.cc:1397] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:36.317502 31633 tablet_bootstrap.cc:492] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: No bootstrap required, opened a new log
I20260810 18:58:36.317582 31633 ts_tablet_manager.cc:1397] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:36.317824 31630 tablet_bootstrap.cc:492] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: No bootstrap required, opened a new log
I20260810 18:58:36.317895 31630 ts_tablet_manager.cc:1397] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:36.317900 31631 raft_consensus.cc:348] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.318029 31631 raft_consensus.cc:374] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:36.317984 31633 raft_consensus.cc:348] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.318060 31631 raft_consensus.cc:729] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Initialized, Role: FOLLOWER
I20260810 18:58:36.318082 31633 raft_consensus.cc:374] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:36.318117 31633 raft_consensus.cc:729] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b7ce52d1ca44e1ae52b84f44ca2ac9, State: Initialized, Role: FOLLOWER
I20260810 18:58:36.318198 31631 consensus_queue.cc:260] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.318235 31633 consensus_queue.cc:260] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.318271 31630 raft_consensus.cc:348] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.318375 31630 raft_consensus.cc:374] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:36.318409 31630 raft_consensus.cc:729] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422918142bbe4ac5871a0f75047f4f4a, State: Initialized, Role: FOLLOWER
I20260810 18:58:36.318519 31633 ts_tablet_manager.cc:1428] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:58:36.318460 31641 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.318472 31642 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.318529 31630 consensus_queue.cc:260] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.318588 31641 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:36.318600 31642 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:36.318784 31640 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.318960 31640 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:36.318797 31630 ts_tablet_manager.cc:1428] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:36.319734 31631 ts_tablet_manager.cc:1428] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:36.353108 31642 raft_consensus.cc:482] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:36.353220 31642 raft_consensus.cc:504] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.354416 31642 leader_election.cc:288] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:36.357951 31629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.354707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:40805 (local address 127.27.147.129:34531)
0810 18:58:36.354845 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.354850 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.354995 (+ 145us) server_negotiation.cc:407] Connection header received
0810 18:58:36.355052 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.355054 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.355093 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.355191 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:36.355464 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.355967 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.356999 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.357383 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.357520 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.357575 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.357762 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.357830 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:58:36.357951 31444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.354569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:34531 (local address 127.27.147.131:40805)
0810 18:58:36.354804 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.354840 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.354952 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.355203 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.355207 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.355224 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:36.355348 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.355353 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.356078 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.356083 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.356866 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.356875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.357511 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.357515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.357634 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.357653 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.357745 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.357816 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153}
I20260810 18:58:36.358383 31665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.354665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:38005 (local address 127.27.147.131:36385)
0810 18:58:36.355051 (+ 386us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.355073 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.355181 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.355437 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.355440 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.355455 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:36.355591 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.355595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.356309 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.356315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.357206 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.357216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.357859 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.357862 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.357968 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.357987 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.358126 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.358203 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"thread_start_us":159,"threads_started":1}
I20260810 18:58:36.358403 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.354790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:36385 (local address 127.27.147.130:38005)
0810 18:58:36.354950 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.354954 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.355153 (+ 199us) server_negotiation.cc:407] Connection header received
0810 18:58:36.355293 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.355301 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.355340 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.355428 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:36.355703 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.356197 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.357328 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.357718 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.357847 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.357906 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.358174 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.358252 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 18:58:36.358676 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140099e5a94465a01eb912ba846237" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a" is_pre_election: true
I20260810 18:58:36.358860 31124 raft_consensus.cc:2454] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:36.359198 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140099e5a94465a01eb912ba846237" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" is_pre_election: true
I20260810 18:58:36.359313 31433 leader_election.cc:302] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422918142bbe4ac5871a0f75047f4f4a, 671621869510425fb0adefb2d0a8ed3c; no voters:
I20260810 18:58:36.359440 31331 raft_consensus.cc:2454] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:36.359643 31642 raft_consensus.cc:2781] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:36.359711 31642 raft_consensus.cc:482] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:36.359771 31642 raft_consensus.cc:3037] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.360755 31642 raft_consensus.cc:504] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.361065 31642 leader_election.cc:288] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Requested vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:36.361423 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140099e5a94465a01eb912ba846237" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9"
I20260810 18:58:36.361434 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "24140099e5a94465a01eb912ba846237" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a"
I20260810 18:58:36.361545 31124 raft_consensus.cc:3037] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.361545 31331 raft_consensus.cc:3037] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.362696 31124 raft_consensus.cc:2454] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:36.362696 31331 raft_consensus.cc:2454] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:36.363109 31431 leader_election.cc:302] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 422918142bbe4ac5871a0f75047f4f4a, 671621869510425fb0adefb2d0a8ed3c, d8b7ce52d1ca44e1ae52b84f44ca2ac9; no voters:
I20260810 18:58:36.363299 31642 raft_consensus.cc:2781] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:36.363551 31642 raft_consensus.cc:686] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [term 1 LEADER]: Becoming Leader. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Running, Role: LEADER
I20260810 18:58:36.363878 31642 consensus_queue.cc:237] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.366109 31666 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: TxnStatusTablet state changed. Reason: New leader 671621869510425fb0adefb2d0a8ed3c. Latest consensus state: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.366231 31666 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:58:36.366515 31668 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:58:36.366762 31668 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:58:36.367534 30889 catalog_manager.cc:5495] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 671621869510425fb0adefb2d0a8ed3c (127.27.147.131). New cstate: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } health_report { overall_health: HEALTHY } } }
I20260810 18:58:36.382735 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.378022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42812 (local address 127.27.147.131:41613)
0810 18:58:36.378141 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.378145 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.378160 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:58:36.378321 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.378325 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.378394 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.378495 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:36.378885 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.379668 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.380927 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.381273 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.381384 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.381838 (+ 454us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.382339 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.382462 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.382549 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.382601 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:58:36.382735 31658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.377913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:41613 (local address 127.0.0.1:42812)
0810 18:58:36.378066 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.378084 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.378182 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.378621 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.378624 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.378633 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:58:36.378736 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.378745 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.379820 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.379823 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.380787 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.380800 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.381483 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.381487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.381580 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.381632 (+ 52us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.382442 (+ 810us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.382460 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.382534 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.382620 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 18:58:36.387681 31331 raft_consensus.cc:1261] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:36.387773 31124 raft_consensus.cc:1261] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:36.388338 31642 consensus_queue.cc:1035] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:36.388543 31666 consensus_queue.cc:1035] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:36.391613 31641 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: TxnStatusTablet state changed. Reason: New leader 671621869510425fb0adefb2d0a8ed3c. Latest consensus state: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.391736 31641 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:36.392282 31640 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: TxnStatusTablet state changed. Reason: New leader 671621869510425fb0adefb2d0a8ed3c. Latest consensus state: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.392423 31640 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:36.396230 31641 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.396340 31641 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:36.396523 31642 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.396625 31642 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:58:36.396759 31642 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.396826 31642 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P 671621869510425fb0adefb2d0a8ed3c: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:58:36.397857 31640 tablet_replica.cc:439] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } } }
I20260810 18:58:36.397966 31640 tablet_replica.cc:442] T 24140099e5a94465a01eb912ba846237 P 422918142bbe4ac5871a0f75047f4f4a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:58:36.398180 31674 mvcc.cc:205] Tried to move back new op lower bound from 7317046543917129728 to 7317046543829061632. Current Snapshot: MvccSnapshot[applied={T|T < 7317046543917129728 or (T in {7317046543917129728})}]
I20260810 18:58:36.398510 31675 mvcc.cc:205] Tried to move back new op lower bound from 7317046543917129728 to 7317046543829061632. Current Snapshot: MvccSnapshot[applied={T|T < 7317046543917129728}]
I20260810 18:58:36.405483 31678 mvcc.cc:205] Tried to move back new op lower bound from 7317046543917129728 to 7317046543829061632. Current Snapshot: MvccSnapshot[applied={T|T < 7317046543917129728}]
I20260810 18:58:36.412434 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.406974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37964 (local address 127.27.147.130:38005)
0810 18:58:36.407083 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.407087 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.407757 (+ 670us) server_negotiation.cc:407] Connection header received
0810 18:58:36.407877 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.407880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.407938 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.408040 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:36.408312 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.409104 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.410590 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.410973 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.411078 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.411340 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.411814 (+ 474us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.412088 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.412232 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.412289 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:58:36.424274 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.420602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:57709 (local address 127.27.147.131:41613)
0810 18:58:36.420717 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.420722 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.420895 (+ 173us) server_negotiation.cc:407] Connection header received
0810 18:58:36.420979 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.420982 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.421035 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.421115 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:36.421379 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.422133 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.423099 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.423453 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.423558 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.423860 (+ 302us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.423939 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.424011 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.424097 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.424148 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:58:36.424284 31692 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.420502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:41613 (local address 127.27.147.130:57709)
0810 18:58:36.420806 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.420819 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.420906 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.421135 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.421137 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.421142 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:58:36.421247 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.421276 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.422274 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.422277 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.422985 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.422996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.423587 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.423590 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.423667 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.423710 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.424028 (+ 318us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.424035 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.424086 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.424150 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":106,"threads_started":1}
I20260810 18:58:36.431368 31124 raft_consensus.cc:1261] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Refusing update from remote peer d8b7ce52d1ca44e1ae52b84f44ca2ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:36.431594 31534 raft_consensus.cc:1261] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Refusing update from remote peer d8b7ce52d1ca44e1ae52b84f44ca2ac9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:36.432310 31646 consensus_queue.cc:1035] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:36.432526 31641 consensus_queue.cc:1035] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:36.454921 31627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.451155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:59445 (local address 127.27.147.130:38005)
0810 18:58:36.451282 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.451287 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.451512 (+ 225us) server_negotiation.cc:407] Connection header received
0810 18:58:36.451610 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.451613 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.451661 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.451746 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:36.452026 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.452689 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.453780 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.454021 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.454142 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.454461 (+ 319us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.454540 (+ 79us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.454624 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.454716 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.454766 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:58:36.454995 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.451036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:38005 (local address 127.27.147.131:59445)
0810 18:58:36.451435 (+ 399us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.451448 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.451540 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.451747 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.451748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.451753 (+ 5us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:58:36.451920 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.451930 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.452842 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.452846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.453654 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.453666 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.454159 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.454163 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.454262 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.454305 (+ 43us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.454632 (+ 327us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.454641 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.454706 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.454786 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":118,"threads_started":1}
I20260810 18:58:36.483297 30888 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:36.483512 30888 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:36.486456 30888 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Sending DeleteTablet for 3 replicas of tablet 8d6a57e3d9614f7e972fdc389b33a94d
I20260810 18:58:36.487262 31093 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6a57e3d9614f7e972fdc389b33a94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:36 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:45700
I20260810 18:58:36.487412 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6a57e3d9614f7e972fdc389b33a94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:36 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:37948
I20260810 18:58:36.487890 31712 tablet_replica.cc:330] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: stopping tablet replica
I20260810 18:58:36.488190 31712 raft_consensus.cc:2229] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:36.488544 31712 raft_consensus.cc:2258] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:36.488663 30888 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:36.489074 30888 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:36.489940 31712 ts_tablet_manager.cc:1905] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:36.491590 31499 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d6a57e3d9614f7e972fdc389b33a94d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:36 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:42804
I20260810 18:58:36.493350 31711 tablet_replica.cc:330] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: stopping tablet replica
I20260810 18:58:36.493716 31711 raft_consensus.cc:2229] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:36.494045 31711 raft_consensus.cc:2258] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:36.495354 31093 tablet_service.cc:1467] Processing CreateTablet for tablet c5a84bea59f9411da80ffcdfccadcfa3 (DEFAULT_TABLE table=test-table [id=f5c0cbd84d70405c856d56557c04d700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:36.495661 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a84bea59f9411da80ffcdfccadcfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:36.495663 31299 tablet_service.cc:1467] Processing CreateTablet for tablet c5a84bea59f9411da80ffcdfccadcfa3 (DEFAULT_TABLE table=test-table [id=f5c0cbd84d70405c856d56557c04d700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:36.496176 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a84bea59f9411da80ffcdfccadcfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:36.498327 31633 tablet_bootstrap.cc:492] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Bootstrap starting.
I20260810 18:58:36.498366 31499 tablet_service.cc:1467] Processing CreateTablet for tablet c5a84bea59f9411da80ffcdfccadcfa3 (DEFAULT_TABLE table=test-table [id=f5c0cbd84d70405c856d56557c04d700]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:36.498627 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c5a84bea59f9411da80ffcdfccadcfa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:36.499337 31633 tablet_bootstrap.cc:654] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:36.499577 31712 ts_tablet_manager.cc:1918] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:36.499658 31712 log.cc:1196] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/8d6a57e3d9614f7e972fdc389b33a94d
I20260810 18:58:36.499997 31712 ts_tablet_manager.cc:1939] T 8d6a57e3d9614f7e972fdc389b33a94d P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting consensus metadata
I20260810 18:58:36.501088 30868 catalog_manager.cc:4841] TS d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005): tablet 8d6a57e3d9614f7e972fdc389b33a94d (table test-table [id=db297cfe3b1541cd9b5b8ca230f8cbe2]) successfully deleted
I20260810 18:58:36.501327 31630 tablet_bootstrap.cc:492] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: Bootstrap starting.
I20260810 18:58:36.501431 31713 tablet_replica.cc:330] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: stopping tablet replica
I20260810 18:58:36.501699 31713 raft_consensus.cc:2229] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:36.501861 31713 raft_consensus.cc:2258] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:36.502254 31630 tablet_bootstrap.cc:654] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:36.502912 31631 tablet_bootstrap.cc:492] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: Bootstrap starting.
I20260810 18:58:36.503052 31713 ts_tablet_manager.cc:1905] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:36.504289 31631 tablet_bootstrap.cc:654] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:36.505792 31713 ts_tablet_manager.cc:1918] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:36.505883 31713 log.cc:1196] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/8d6a57e3d9614f7e972fdc389b33a94d
I20260810 18:58:36.506210 31713 ts_tablet_manager.cc:1939] T 8d6a57e3d9614f7e972fdc389b33a94d P 671621869510425fb0adefb2d0a8ed3c: Deleting consensus metadata
I20260810 18:58:36.507161 31711 ts_tablet_manager.cc:1905] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:36.507318 30871 catalog_manager.cc:4841] TS 671621869510425fb0adefb2d0a8ed3c (127.27.147.131:41613): tablet 8d6a57e3d9614f7e972fdc389b33a94d (table test-table [id=db297cfe3b1541cd9b5b8ca230f8cbe2]) successfully deleted
I20260810 18:58:36.510291 31711 ts_tablet_manager.cc:1918] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:36.510401 31711 log.cc:1196] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/8d6a57e3d9614f7e972fdc389b33a94d
I20260810 18:58:36.510710 31711 ts_tablet_manager.cc:1939] T 8d6a57e3d9614f7e972fdc389b33a94d P 422918142bbe4ac5871a0f75047f4f4a: Deleting consensus metadata
I20260810 18:58:36.511909 30871 catalog_manager.cc:4841] TS 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531): tablet 8d6a57e3d9614f7e972fdc389b33a94d (table test-table [id=db297cfe3b1541cd9b5b8ca230f8cbe2]) successfully deleted
I20260810 18:58:36.513041 31631 tablet_bootstrap.cc:492] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: No bootstrap required, opened a new log
I20260810 18:58:36.513043 31630 tablet_bootstrap.cc:492] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: No bootstrap required, opened a new log
I20260810 18:58:36.513121 31631 ts_tablet_manager.cc:1397] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260810 18:58:36.513128 31630 ts_tablet_manager.cc:1397] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260810 18:58:36.513530 31631 raft_consensus.cc:348] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.513571 31630 raft_consensus.cc:348] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.513643 31631 raft_consensus.cc:374] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:36.513674 31631 raft_consensus.cc:729] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Initialized, Role: FOLLOWER
I20260810 18:58:36.513679 31630 raft_consensus.cc:374] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:36.513712 31630 raft_consensus.cc:729] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422918142bbe4ac5871a0f75047f4f4a, State: Initialized, Role: FOLLOWER
I20260810 18:58:36.513796 31631 consensus_queue.cc:260] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.513827 31630 consensus_queue.cc:260] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.514034 31631 ts_tablet_manager.cc:1428] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:36.514086 31630 ts_tablet_manager.cc:1428] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:36.515574 31633 tablet_bootstrap.cc:492] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: No bootstrap required, opened a new log
I20260810 18:58:36.515677 31633 ts_tablet_manager.cc:1397] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.002s
I20260810 18:58:36.516131 31633 raft_consensus.cc:348] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.516254 31633 raft_consensus.cc:374] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:36.516291 31633 raft_consensus.cc:729] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b7ce52d1ca44e1ae52b84f44ca2ac9, State: Initialized, Role: FOLLOWER
I20260810 18:58:36.516428 31633 consensus_queue.cc:260] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.516693 31633 ts_tablet_manager.cc:1428] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:58:36.520880 31685 raft_consensus.cc:482] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:36.520984 31685 raft_consensus.cc:504] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.521397 31685 leader_election.cc:288] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:36.521731 31122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a84bea59f9411da80ffcdfccadcfa3" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a" is_pre_election: true
I20260810 18:58:36.521874 31122 raft_consensus.cc:2454] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:36.522466 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a84bea59f9411da80ffcdfccadcfa3" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" is_pre_election: true
I20260810 18:58:36.522603 31331 raft_consensus.cc:2454] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:36.522871 31433 leader_election.cc:302] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422918142bbe4ac5871a0f75047f4f4a, 671621869510425fb0adefb2d0a8ed3c; no voters:
I20260810 18:58:36.523089 31685 raft_consensus.cc:2781] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:36.523155 31685 raft_consensus.cc:482] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:36.523202 31685 raft_consensus.cc:3037] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.524420 31685 raft_consensus.cc:504] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.524793 31685 leader_election.cc:288] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Requested vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:36.525170 31122 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a84bea59f9411da80ffcdfccadcfa3" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a"
I20260810 18:58:36.525270 31122 raft_consensus.cc:3037] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.525238 31331 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c5a84bea59f9411da80ffcdfccadcfa3" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9"
I20260810 18:58:36.525363 31331 raft_consensus.cc:3037] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:36.526381 31122 raft_consensus.cc:2454] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:36.526547 31331 raft_consensus.cc:2454] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:36.526845 31433 leader_election.cc:302] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422918142bbe4ac5871a0f75047f4f4a, 671621869510425fb0adefb2d0a8ed3c; no voters:
I20260810 18:58:36.527033 31685 raft_consensus.cc:2781] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:36.527138 31685 raft_consensus.cc:686] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 1 LEADER]: Becoming Leader. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Running, Role: LEADER
I20260810 18:58:36.527267 31685 consensus_queue.cc:237] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:36.528872 30888 catalog_manager.cc:5495] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 671621869510425fb0adefb2d0a8ed3c (127.27.147.131). New cstate: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } health_report { overall_health: HEALTHY } } }
I20260810 18:58:36.548244 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.544353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.27.147.131:41613)
0810 18:58:36.544460 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.544464 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.544528 (+ 64us) server_negotiation.cc:407] Connection header received
0810 18:58:36.544678 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.544681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.544733 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.544812 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:36.545076 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.545879 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.547056 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.547404 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.547511 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.547775 (+ 264us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.547859 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.547929 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.548058 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.548106 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 18:58:36.558274 31625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.554582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:46661 (local address 127.27.147.131:41613)
0810 18:58:36.554676 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:36.554680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:36.554720 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:58:36.554874 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:36.554877 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:36.554922 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:36.554997 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:36.555218 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.555954 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.557079 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.557412 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.557513 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.557756 (+ 243us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.557834 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.557904 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:36.557999 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 18:58:36.558105 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":34}
I20260810 18:58:36.558338 31704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:36.554510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:41613 (local address 127.27.147.131:46661)
0810 18:58:36.554669 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:36.554682 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:36.554758 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:36.555020 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:36.555022 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:36.555029 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:58:36.555116 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.555121 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.556058 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.556061 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.556962 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:36.556970 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:36.557525 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:36.557529 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:36.557615 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:36.557644 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:58:36.557922 (+ 278us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:58:36.557928 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:58:36.557994 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:58:36.558118 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":1}
I20260810 18:58:36.564311 31331 raft_consensus.cc:1261] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:36.564568 31122 raft_consensus.cc:1261] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:36.564725 31642 consensus_queue.cc:1035] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:36.565002 31684 consensus_queue.cc:1035] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:58:36.580018 31674 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:58:38.139113 30889 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:38.139266 30889 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56614:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:38.141479 30889 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Sending DeleteTablet for 3 replicas of tablet c5a84bea59f9411da80ffcdfccadcfa3
I20260810 18:58:38.142045 31093 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a84bea59f9411da80ffcdfccadcfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:38 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:45700
I20260810 18:58:38.142083 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a84bea59f9411da80ffcdfccadcfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:38 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:37948
I20260810 18:58:38.142076 31499 tablet_service.cc:1514] Processing DeleteTablet for tablet c5a84bea59f9411da80ffcdfccadcfa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:38 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:42804
I20260810 18:58:38.142477 31753 tablet_replica.cc:330] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: stopping tablet replica
I20260810 18:58:38.142628 31753 raft_consensus.cc:2229] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:38.142961 31753 raft_consensus.cc:2258] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:38.142997 31751 tablet_replica.cc:330] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: stopping tablet replica
I20260810 18:58:38.143218 31751 raft_consensus.cc:2229] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:38.143224 31752 tablet_replica.cc:330] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: stopping tablet replica
I20260810 18:58:38.143383 31752 raft_consensus.cc:2229] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:38.143376 31751 raft_consensus.cc:2258] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:38.143522 31752 raft_consensus.cc:2258] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:38.144273 31753 ts_tablet_manager.cc:1905] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:38.144275 31751 ts_tablet_manager.cc:1905] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:38.146636 31753 ts_tablet_manager.cc:1918] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:58:38.146690 31753 log.cc:1196] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/c5a84bea59f9411da80ffcdfccadcfa3
I20260810 18:58:38.147017 31753 ts_tablet_manager.cc:1939] T c5a84bea59f9411da80ffcdfccadcfa3 P 671621869510425fb0adefb2d0a8ed3c: Deleting consensus metadata
I20260810 18:58:38.147464 31751 ts_tablet_manager.cc:1918] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:58:38.147554 31751 log.cc:1196] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/c5a84bea59f9411da80ffcdfccadcfa3
I20260810 18:58:38.147739 30871 catalog_manager.cc:4841] TS 671621869510425fb0adefb2d0a8ed3c (127.27.147.131:41613): tablet c5a84bea59f9411da80ffcdfccadcfa3 (table test-table [id=f5c0cbd84d70405c856d56557c04d700]) successfully deleted
I20260810 18:58:38.147845 31751 ts_tablet_manager.cc:1939] T c5a84bea59f9411da80ffcdfccadcfa3 P 422918142bbe4ac5871a0f75047f4f4a: Deleting consensus metadata
I20260810 18:58:38.148010 31752 ts_tablet_manager.cc:1905] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:38.148518 30871 catalog_manager.cc:4841] TS 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531): tablet c5a84bea59f9411da80ffcdfccadcfa3 (table test-table [id=f5c0cbd84d70405c856d56557c04d700]) successfully deleted
I20260810 18:58:38.151294 31752 ts_tablet_manager.cc:1918] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:58:38.151379 31752 log.cc:1196] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/c5a84bea59f9411da80ffcdfccadcfa3
I20260810 18:58:38.151628 31752 ts_tablet_manager.cc:1939] T c5a84bea59f9411da80ffcdfccadcfa3 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting consensus metadata
I20260810 18:58:38.152274 30868 catalog_manager.cc:4841] TS d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005): tablet c5a84bea59f9411da80ffcdfccadcfa3 (table test-table [id=f5c0cbd84d70405c856d56557c04d700]) successfully deleted
Aug 10 18:58:38 dist-test-slave-znh6 krb5kdc[30822](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:58:38 dist-test-slave-znh6 krb5kdc[30822](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388318, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:38.173226 31764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:38.164024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56626 (local address 127.27.147.190:35783)
0810 18:58:38.164271 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:38.164278 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:38.164345 (+ 67us) server_negotiation.cc:407] Connection header received
0810 18:58:38.164454 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:38.164456 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:38.164501 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:38.164578 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:38.165263 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:38.165904 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:38.166750 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:38.167060 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:38.167142 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:38.169702 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:38.169705 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:38.169708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:38.169731 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:38.171337 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:38.171809 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:38.171817 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:38.171818 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:38.171901 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:38.172181 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:38.172184 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:38.172186 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:38.172501 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:38.172653 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:38.172909 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 18:58:38.173040 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
I20260810 18:58:38.176513 30889 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:38.177050 30889 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:38.182380 31093 tablet_service.cc:1467] Processing CreateTablet for tablet d2929c14e3bc4d35af4e518469a5d6e7 (DEFAULT_TABLE table=test-table [id=c1565a4ba1284984b21ed7e2ec864691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:38.182359 31299 tablet_service.cc:1467] Processing CreateTablet for tablet d2929c14e3bc4d35af4e518469a5d6e7 (DEFAULT_TABLE table=test-table [id=c1565a4ba1284984b21ed7e2ec864691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:38.182490 31499 tablet_service.cc:1467] Processing CreateTablet for tablet d2929c14e3bc4d35af4e518469a5d6e7 (DEFAULT_TABLE table=test-table [id=c1565a4ba1284984b21ed7e2ec864691]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:38.182710 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2929c14e3bc4d35af4e518469a5d6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:38.182715 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2929c14e3bc4d35af4e518469a5d6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:38.182752 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2929c14e3bc4d35af4e518469a5d6e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:38.184850 31766 tablet_bootstrap.cc:492] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: Bootstrap starting.
I20260810 18:58:38.185472 31767 tablet_bootstrap.cc:492] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: Bootstrap starting.
I20260810 18:58:38.185771 31766 tablet_bootstrap.cc:654] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:38.186398 31767 tablet_bootstrap.cc:654] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:38.186605 31768 tablet_bootstrap.cc:492] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Bootstrap starting.
I20260810 18:58:38.187392 31766 tablet_bootstrap.cc:492] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: No bootstrap required, opened a new log
I20260810 18:58:38.187489 31766 ts_tablet_manager.cc:1397] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:38.187608 31768 tablet_bootstrap.cc:654] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:38.187829 31767 tablet_bootstrap.cc:492] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: No bootstrap required, opened a new log
I20260810 18:58:38.187989 31767 ts_tablet_manager.cc:1397] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:38.188105 31766 raft_consensus.cc:348] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.188315 31766 raft_consensus.cc:374] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:38.188398 31766 raft_consensus.cc:729] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422918142bbe4ac5871a0f75047f4f4a, State: Initialized, Role: FOLLOWER
I20260810 18:58:38.188561 31766 consensus_queue.cc:260] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.188565 31767 raft_consensus.cc:348] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.188694 31767 raft_consensus.cc:374] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:38.188751 31767 raft_consensus.cc:729] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Initialized, Role: FOLLOWER
I20260810 18:58:38.188894 31768 tablet_bootstrap.cc:492] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: No bootstrap required, opened a new log
I20260810 18:58:38.188917 31767 consensus_queue.cc:260] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.188980 31768 ts_tablet_manager.cc:1397] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:38.189005 31766 ts_tablet_manager.cc:1428] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:58:38.189198 31767 ts_tablet_manager.cc:1428] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:38.189432 31768 raft_consensus.cc:348] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.189553 31768 raft_consensus.cc:374] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:38.189595 31768 raft_consensus.cc:729] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b7ce52d1ca44e1ae52b84f44ca2ac9, State: Initialized, Role: FOLLOWER
I20260810 18:58:38.189720 31768 consensus_queue.cc:260] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.190129 31768 ts_tablet_manager.cc:1428] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:38.362870 31726 raft_consensus.cc:482] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:38.363039 31726 raft_consensus.cc:504] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.363533 31726 leader_election.cc:288] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:38.364022 31329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2929c14e3bc4d35af4e518469a5d6e7" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" is_pre_election: true
I20260810 18:58:38.364022 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2929c14e3bc4d35af4e518469a5d6e7" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a" is_pre_election: true
I20260810 18:58:38.364161 31329 raft_consensus.cc:2454] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:38.364197 31124 raft_consensus.cc:2454] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:38.364526 31431 leader_election.cc:302] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671621869510425fb0adefb2d0a8ed3c, d8b7ce52d1ca44e1ae52b84f44ca2ac9; no voters:
I20260810 18:58:38.364713 31726 raft_consensus.cc:2781] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:38.364780 31726 raft_consensus.cc:482] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:38.364807 31726 raft_consensus.cc:3037] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:38.365944 31726 raft_consensus.cc:504] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.366341 31726 leader_election.cc:288] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Requested vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:38.366751 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2929c14e3bc4d35af4e518469a5d6e7" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a"
I20260810 18:58:38.366770 31329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2929c14e3bc4d35af4e518469a5d6e7" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9"
I20260810 18:58:38.366899 31124 raft_consensus.cc:3037] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:38.366951 31329 raft_consensus.cc:3037] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:38.367806 31124 raft_consensus.cc:2454] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:38.368017 31329 raft_consensus.cc:2454] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:38.368112 31433 leader_election.cc:302] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 422918142bbe4ac5871a0f75047f4f4a, 671621869510425fb0adefb2d0a8ed3c; no voters:
I20260810 18:58:38.368271 31726 raft_consensus.cc:2781] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:38.368361 31726 raft_consensus.cc:686] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 1 LEADER]: Becoming Leader. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Running, Role: LEADER
I20260810 18:58:38.368499 31726 consensus_queue.cc:237] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.370400 30890 catalog_manager.cc:5495] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 671621869510425fb0adefb2d0a8ed3c (127.27.147.131). New cstate: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } health_report { overall_health: HEALTHY } } }
I20260810 18:58:38.385210 31782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:38.382143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42826 (local address 127.27.147.131:41613)
0810 18:58:38.382450 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:38.382457 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:38.382469 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:58:38.382505 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:38.382507 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:38.382549 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:38.382623 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:38.382894 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:38.383449 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:38.384273 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:38.384507 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:38.384582 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:38.384824 (+ 242us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:38.384893 (+ 69us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:38.384954 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:38.385052 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 18:58:38.385093 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":158,"threads_started":1}
I20260810 18:58:38.391685 31329 raft_consensus.cc:1261] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:38.391968 31124 raft_consensus.cc:1261] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:38.392064 31734 consensus_queue.cc:1035] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:38.392386 31734 consensus_queue.cc:1035] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:38.426038 30890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:38.426182 30890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:38.428056 30890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Sending DeleteTablet for 3 replicas of tablet d2929c14e3bc4d35af4e518469a5d6e7
I20260810 18:58:38.428638 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet d2929c14e3bc4d35af4e518469a5d6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:38 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:37948
I20260810 18:58:38.428647 31093 tablet_service.cc:1514] Processing DeleteTablet for tablet d2929c14e3bc4d35af4e518469a5d6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:38 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:45700
I20260810 18:58:38.428828 31752 tablet_replica.cc:330] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: stopping tablet replica
I20260810 18:58:38.428831 31499 tablet_service.cc:1514] Processing DeleteTablet for tablet d2929c14e3bc4d35af4e518469a5d6e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:38 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:42804
I20260810 18:58:38.428970 31752 raft_consensus.cc:2229] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:38.428990 31753 tablet_replica.cc:330] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: stopping tablet replica
I20260810 18:58:38.428999 31751 tablet_replica.cc:330] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: stopping tablet replica
I20260810 18:58:38.429087 31752 raft_consensus.cc:2258] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:38.429097 31751 raft_consensus.cc:2229] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:38.429101 31753 raft_consensus.cc:2229] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:38.429193 31751 raft_consensus.cc:2258] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:38.429286 31753 raft_consensus.cc:2258] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:38.429533 30890 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56626:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:38.429947 30890 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:38.429973 31752 ts_tablet_manager.cc:1905] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:38.430089 31753 ts_tablet_manager.cc:1905] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:38.432626 31752 ts_tablet_manager.cc:1918] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:38.432626 31753 ts_tablet_manager.cc:1918] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:38.432713 31752 log.cc:1196] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/d2929c14e3bc4d35af4e518469a5d6e7
I20260810 18:58:38.432713 31753 log.cc:1196] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/d2929c14e3bc4d35af4e518469a5d6e7
I20260810 18:58:38.432982 31752 ts_tablet_manager.cc:1939] T d2929c14e3bc4d35af4e518469a5d6e7 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting consensus metadata
I20260810 18:58:38.432983 31753 ts_tablet_manager.cc:1939] T d2929c14e3bc4d35af4e518469a5d6e7 P 671621869510425fb0adefb2d0a8ed3c: Deleting consensus metadata
I20260810 18:58:38.433725 30871 catalog_manager.cc:4841] TS 671621869510425fb0adefb2d0a8ed3c (127.27.147.131:41613): tablet d2929c14e3bc4d35af4e518469a5d6e7 (table test-table [id=c1565a4ba1284984b21ed7e2ec864691]) successfully deleted
I20260810 18:58:38.433774 30868 catalog_manager.cc:4841] TS d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005): tablet d2929c14e3bc4d35af4e518469a5d6e7 (table test-table [id=c1565a4ba1284984b21ed7e2ec864691]) successfully deleted
I20260810 18:58:38.434109 31751 ts_tablet_manager.cc:1905] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:38.436244 31093 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7f4a439ccb47d9bb04440e4d581711 (DEFAULT_TABLE table=test-table [id=a1ac1e4c025b467d983bf6a1f87d7ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:38.436353 31299 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7f4a439ccb47d9bb04440e4d581711 (DEFAULT_TABLE table=test-table [id=a1ac1e4c025b467d983bf6a1f87d7ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:38.436439 31499 tablet_service.cc:1467] Processing CreateTablet for tablet 8f7f4a439ccb47d9bb04440e4d581711 (DEFAULT_TABLE table=test-table [id=a1ac1e4c025b467d983bf6a1f87d7ca6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:38.436508 31093 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7f4a439ccb47d9bb04440e4d581711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:38.436611 31299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7f4a439ccb47d9bb04440e4d581711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:38.436652 31499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f7f4a439ccb47d9bb04440e4d581711. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:38.436717 31751 ts_tablet_manager.cc:1918] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:58:38.436785 31751 log.cc:1196] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/d2929c14e3bc4d35af4e518469a5d6e7
I20260810 18:58:38.437049 31751 ts_tablet_manager.cc:1939] T d2929c14e3bc4d35af4e518469a5d6e7 P 422918142bbe4ac5871a0f75047f4f4a: Deleting consensus metadata
I20260810 18:58:38.437757 30871 catalog_manager.cc:4841] TS 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531): tablet d2929c14e3bc4d35af4e518469a5d6e7 (table test-table [id=c1565a4ba1284984b21ed7e2ec864691]) successfully deleted
I20260810 18:58:38.438647 31766 tablet_bootstrap.cc:492] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a: Bootstrap starting.
I20260810 18:58:38.438889 31768 tablet_bootstrap.cc:492] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Bootstrap starting.
I20260810 18:58:38.438900 31767 tablet_bootstrap.cc:492] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: Bootstrap starting.
I20260810 18:58:38.439572 31768 tablet_bootstrap.cc:654] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:38.439642 31766 tablet_bootstrap.cc:654] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:38.439642 31767 tablet_bootstrap.cc:654] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:38.440691 31768 tablet_bootstrap.cc:492] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: No bootstrap required, opened a new log
I20260810 18:58:38.440788 31768 ts_tablet_manager.cc:1397] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:38.441161 31768 raft_consensus.cc:348] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.441262 31767 tablet_bootstrap.cc:492] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: No bootstrap required, opened a new log
I20260810 18:58:38.441246 31768 raft_consensus.cc:374] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:38.441304 31768 raft_consensus.cc:729] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8b7ce52d1ca44e1ae52b84f44ca2ac9, State: Initialized, Role: FOLLOWER
I20260810 18:58:38.441334 31767 ts_tablet_manager.cc:1397] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:38.441445 31768 consensus_queue.cc:260] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.441702 31768 ts_tablet_manager.cc:1428] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:58:38.441716 31767 raft_consensus.cc:348] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.441802 31767 raft_consensus.cc:374] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:38.441823 31767 raft_consensus.cc:729] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Initialized, Role: FOLLOWER
I20260810 18:58:38.441962 31767 consensus_queue.cc:260] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.442252 31767 ts_tablet_manager.cc:1428] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:58:38.442255 31766 tablet_bootstrap.cc:492] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a: No bootstrap required, opened a new log
I20260810 18:58:38.442346 31766 ts_tablet_manager.cc:1397] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260810 18:58:38.442799 31766 raft_consensus.cc:348] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.442950 31766 raft_consensus.cc:374] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:38.442974 31766 raft_consensus.cc:729] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 422918142bbe4ac5871a0f75047f4f4a, State: Initialized, Role: FOLLOWER
I20260810 18:58:38.443094 31766 consensus_queue.cc:260] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.443392 31766 ts_tablet_manager.cc:1428] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:58:38.658186 31666 raft_consensus.cc:482] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:38.658437 31666 raft_consensus.cc:504] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.659000 31666 leader_election.cc:288] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:38.659518 31329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7f4a439ccb47d9bb04440e4d581711" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" is_pre_election: true
I20260810 18:58:38.659518 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7f4a439ccb47d9bb04440e4d581711" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a" is_pre_election: true
I20260810 18:58:38.659682 31124 raft_consensus.cc:2454] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:38.659682 31329 raft_consensus.cc:2454] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 0.
I20260810 18:58:38.660019 31431 leader_election.cc:302] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 422918142bbe4ac5871a0f75047f4f4a, 671621869510425fb0adefb2d0a8ed3c, d8b7ce52d1ca44e1ae52b84f44ca2ac9; no voters:
I20260810 18:58:38.660230 31666 raft_consensus.cc:2781] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:38.660315 31666 raft_consensus.cc:482] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:38.660356 31666 raft_consensus.cc:3037] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:38.661473 31666 raft_consensus.cc:504] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.661892 31666 leader_election.cc:288] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Requested vote from peers 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531), d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005)
I20260810 18:58:38.662228 31124 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7f4a439ccb47d9bb04440e4d581711" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "422918142bbe4ac5871a0f75047f4f4a"
I20260810 18:58:38.662298 31329 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f7f4a439ccb47d9bb04440e4d581711" candidate_uuid: "671621869510425fb0adefb2d0a8ed3c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9"
I20260810 18:58:38.662398 31124 raft_consensus.cc:3037] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:38.662423 31329 raft_consensus.cc:3037] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:38.663627 31329 raft_consensus.cc:2454] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:38.663627 31124 raft_consensus.cc:2454] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 671621869510425fb0adefb2d0a8ed3c in term 1.
I20260810 18:58:38.663923 31431 leader_election.cc:302] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 671621869510425fb0adefb2d0a8ed3c, d8b7ce52d1ca44e1ae52b84f44ca2ac9; no voters:
I20260810 18:58:38.664124 31666 raft_consensus.cc:2781] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:38.664260 31666 raft_consensus.cc:686] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 1 LEADER]: Becoming Leader. State: Replica: 671621869510425fb0adefb2d0a8ed3c, State: Running, Role: LEADER
I20260810 18:58:38.664418 31666 consensus_queue.cc:237] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } }
I20260810 18:58:38.666266 30889 catalog_manager.cc:5495] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c reported cstate change: term changed from 0 to 1, leader changed from <none> to 671621869510425fb0adefb2d0a8ed3c (127.27.147.131). New cstate: current_term: 1 leader_uuid: "671621869510425fb0adefb2d0a8ed3c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "671621869510425fb0adefb2d0a8ed3c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 41613 } health_report { overall_health: HEALTHY } } }
I20260810 18:58:38.689402 31124 raft_consensus.cc:1261] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:38.689402 31329 raft_consensus.cc:1261] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Refusing update from remote peer 671621869510425fb0adefb2d0a8ed3c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:38.689795 31726 consensus_queue.cc:1035] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "422918142bbe4ac5871a0f75047f4f4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:38.689958 31745 consensus_queue.cc:1035] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8b7ce52d1ca44e1ae52b84f44ca2ac9" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 38005 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:58:38.701871 31724 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 18:58:40.252096 30890 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:40.252250 30890 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56626:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:40.254707 30890 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9bdf65ef5ffa475badaac9e319e1013a: Sending DeleteTablet for 3 replicas of tablet 8f7f4a439ccb47d9bb04440e4d581711
I20260810 18:58:40.255322 31093 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7f4a439ccb47d9bb04440e4d581711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:40 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:45700
I20260810 18:58:40.255391 31499 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7f4a439ccb47d9bb04440e4d581711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:40 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:42804
I20260810 18:58:40.255426 31299 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f7f4a439ccb47d9bb04440e4d581711 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:40 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:37948
I20260810 18:58:40.255800 31826 tablet_replica.cc:330] T 8f7f4a439ccb47d9bb04440e4d581711 P 422918142bbe4ac5871a0f75047f4f4a: stopping tablet replica
I20260810 18:58:40.255856 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30989
I20260810 18:58:40.271726 31829 tablet_replica.cc:330] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: stopping tablet replica
I20260810 18:58:40.271952 31829 raft_consensus.cc:2229] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:40.272131 31829 raft_consensus.cc:2258] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:40.272430 31827 tablet_replica.cc:330] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: stopping tablet replica
I20260810 18:58:40.272576 31827 raft_consensus.cc:2229] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:40.272825 31827 raft_consensus.cc:2258] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:40.273461 31829 ts_tablet_manager.cc:1905] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:40.275785 31829 ts_tablet_manager.cc:1918] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:58:40.275888 31829 log.cc:1196] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/8f7f4a439ccb47d9bb04440e4d581711
I20260810 18:58:40.276170 31829 ts_tablet_manager.cc:1939] T 8f7f4a439ccb47d9bb04440e4d581711 P d8b7ce52d1ca44e1ae52b84f44ca2ac9: Deleting consensus metadata
I20260810 18:58:40.277032 30868 catalog_manager.cc:4841] TS d8b7ce52d1ca44e1ae52b84f44ca2ac9 (127.27.147.130:38005): tablet 8f7f4a439ccb47d9bb04440e4d581711 (table test-table [id=a1ac1e4c025b467d983bf6a1f87d7ca6]) successfully deleted
I20260810 18:58:40.277323 31827 ts_tablet_manager.cc:1905] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 18:58:40.278553 30871 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:34531 (error 108)
I20260810 18:58:40.279825 31827 ts_tablet_manager.cc:1918] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 18:58:40.279901 31831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:40.279422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:34531 (local address 127.0.0.1:45702)
0810 18:58:40.279677 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:40.279767 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:34531: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
I20260810 18:58:40.279930 31827 log.cc:1196] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/8f7f4a439ccb47d9bb04440e4d581711
I20260810 18:58:40.280032 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31209
W20260810 18:58:40.280056 30871 catalog_manager.cc:4584] TS 422918142bbe4ac5871a0f75047f4f4a (127.27.147.129:34531): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f7f4a439ccb47d9bb04440e4d581711: Network error: Client connection negotiation failed: client connection to 127.27.147.129:34531: connect: Connection refused (error 111)
I20260810 18:58:40.280259 31827 ts_tablet_manager.cc:1939] T 8f7f4a439ccb47d9bb04440e4d581711 P 671621869510425fb0adefb2d0a8ed3c: Deleting consensus metadata
I20260810 18:58:40.290568 30871 catalog_manager.cc:4841] TS 671621869510425fb0adefb2d0a8ed3c (127.27.147.131:41613): tablet 8f7f4a439ccb47d9bb04440e4d581711 (table test-table [id=a1ac1e4c025b467d983bf6a1f87d7ca6]) successfully deleted
I20260810 18:58:40.297879 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31405
I20260810 18:58:40.310889 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30845
2026-08-10T18:58:40Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (6759 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:40 dist-test-slave-znh6 krb5kdc[31836](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:40 dist-test-slave-znh6 krb5kdc[31836](info): set up 2 sockets
Aug 10 18:58:40 dist-test-slave-znh6 krb5kdc[31836](info): commencing operation
krb5kdc: starting...
W20260810 18:58:41.576232 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:41 dist-test-slave-znh6 krb5kdc[31836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:41.701942 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41279
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:41279
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:41.816393 31854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:41.816704 31854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:41.816780 31854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:41.820719 31854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:41.820811 31854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:41.820852 31854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:41.820912 31854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:41.820952 31854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:41.826069 31854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:41279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31854
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:41.827124 31854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:41.828143 31854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:41.837028 31862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:41.837028 31860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:41.837095 31859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:41.837168 31854 server_base.cc:1034] running on GCE node
I20260810 18:58:41.837989 31854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:41.839118 31854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:41.840283 31854 hybrid_clock.cc:648] HybridClock initialized: now 1786388321840254 us; error 42 us; skew 500 ppm
Aug 10 18:58:41 dist-test-slave-znh6 krb5kdc[31836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388321, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:41.846740 31854 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:41.847710 31854 webserver.cc:460] Webserver started at http://127.27.147.190:35735/ using document root <none> and password file <none>
I20260810 18:58:41.848299 31854 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:41.848362 31854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:41.848608 31854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:41.850718 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "2d17e96bdb1147eeab9a34c37daef311"
format_stamp: "Formatted at 2026-08-10 18:58:41 on dist-test-slave-znh6"
server_key: "b696a8f459cf80f8edd7ba09383e74cd"
server_key_iv: "09d576c74ba297e4a664bd6c32543a3a"
server_key_version: "encryptionkey@0"
I20260810 18:58:41.851368 31854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "2d17e96bdb1147eeab9a34c37daef311"
format_stamp: "Formatted at 2026-08-10 18:58:41 on dist-test-slave-znh6"
server_key: "b696a8f459cf80f8edd7ba09383e74cd"
server_key_iv: "09d576c74ba297e4a664bd6c32543a3a"
server_key_version: "encryptionkey@0"
I20260810 18:58:41.855212 31854 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 18:58:41.857687 31871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:41.859021 31854 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:41.859153 31854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "2d17e96bdb1147eeab9a34c37daef311"
format_stamp: "Formatted at 2026-08-10 18:58:41 on dist-test-slave-znh6"
server_key: "b696a8f459cf80f8edd7ba09383e74cd"
server_key_iv: "09d576c74ba297e4a664bd6c32543a3a"
server_key_version: "encryptionkey@0"
I20260810 18:58:41.859259 31854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:41.875540 31854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:41.878557 31854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:41.878762 31854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:41.886322 31854 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:41279
I20260810 18:58:41.886348 31948 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:41279 every 8 connection(s)
I20260810 18:58:41.887298 31854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:41.888370 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31854
I20260810 18:58:41.888487 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:41.888823 28238 external_mini_cluster.cc:1428] Setting key 9cbc82de73e5aad2c7fd902312145ee7
I20260810 18:58:41.890725 31950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:41.896306 31950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311: Bootstrap starting.
Aug 10 18:58:41 dist-test-slave-znh6 krb5kdc[31836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388321, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:41.899042 31950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:41.899926 31950 log.cc:824] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:41.902287 31950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311: No bootstrap required, opened a new log
I20260810 18:58:41.903625 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:41.890281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34702 (local address 127.27.147.190:41279)
0810 18:58:41.890718 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:41.890745 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:41.890808 (+ 63us) server_negotiation.cc:407] Connection header received
0810 18:58:41.891501 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:41.891518 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:41.891834 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:41.892184 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:41.893020 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:41.894031 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:41.894956 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:41.895422 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:41.895607 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:41.898392 (+ 2785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:41.898397 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:41.898410 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:41.898439 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:41.900233 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:41.900718 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:41.900724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:41.900750 (+ 26us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:41.900897 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:41.901251 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:41.901253 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:41.901254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:41.901853 (+ 599us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:41.902059 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:41.902351 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 18:58:41.902795 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":136,"threads_started":1}
I20260810 18:58:41.906352 31950 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d17e96bdb1147eeab9a34c37daef311" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41279 } }
I20260810 18:58:41.906555 31950 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:41.906634 31950 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2d17e96bdb1147eeab9a34c37daef311, State: Initialized, Role: FOLLOWER
I20260810 18:58:41.907117 31950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d17e96bdb1147eeab9a34c37daef311" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41279 } }
I20260810 18:58:41.907280 31950 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:41.907372 31950 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:41.907478 31950 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:41.908324 31950 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d17e96bdb1147eeab9a34c37daef311" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41279 } }
I20260810 18:58:41.908664 31950 leader_election.cc:302] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2d17e96bdb1147eeab9a34c37daef311; no voters:
I20260810 18:58:41.908967 31950 leader_election.cc:288] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:41.909111 31957 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:41.909363 31957 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [term 1 LEADER]: Becoming Leader. State: Replica: 2d17e96bdb1147eeab9a34c37daef311, State: Running, Role: LEADER
I20260810 18:58:41.909698 31957 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d17e96bdb1147eeab9a34c37daef311" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41279 } }
I20260810 18:58:41.910177 31950 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:41.911413 31960 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2d17e96bdb1147eeab9a34c37daef311. Latest consensus state: current_term: 1 leader_uuid: "2d17e96bdb1147eeab9a34c37daef311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d17e96bdb1147eeab9a34c37daef311" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41279 } } }
I20260810 18:58:41.911550 31960 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:41.911759 31958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2d17e96bdb1147eeab9a34c37daef311" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2d17e96bdb1147eeab9a34c37daef311" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41279 } } }
I20260810 18:58:41.911881 31958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:41.911986 31974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:41.915068 31974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:41.920271 31974 catalog_manager.cc:1347] Generated new cluster ID: 9a6c0a64b1ed4a96a15b441a356f1d5d
I20260810 18:58:41.920337 31974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:41.933522 31974 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:41.934433 31974 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:41.944396 31974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2d17e96bdb1147eeab9a34c37daef311: Generated new TSK 0
I20260810 18:58:41.945082 31974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:41.982028 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41279
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:42.119946 31995 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:42.120208 31995 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:42.120251 31995 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:42.124240 31995 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:42.124300 31995 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:42.124354 31995 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:42.129500 31995 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31995
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:42.130690 31995 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:42.131804 31995 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:42.141575 32006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:42.141722 32004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:42.141575 32003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:42.141636 31995 server_base.cc:1034] running on GCE node
I20260810 18:58:42.142227 31995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:42.142918 31995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:42.144119 31995 hybrid_clock.cc:648] HybridClock initialized: now 1786388322144089 us; error 52 us; skew 500 ppm
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[31836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:42.146975 31995 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:42.147915 31995 webserver.cc:460] Webserver started at http://127.27.147.129:38995/ using document root <none> and password file <none>
I20260810 18:58:42.148495 31995 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:42.148588 31995 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:42.148833 31995 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:42.150995 31995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "16d6ab2428014b829cad08c3a3919e6b"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "528f2d22e637f0d0fc171660e0d6d367"
server_key_iv: "d81de71ec1db232dc56bf02d5ecfa541"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.151588 31995 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "16d6ab2428014b829cad08c3a3919e6b"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "528f2d22e637f0d0fc171660e0d6d367"
server_key_iv: "d81de71ec1db232dc56bf02d5ecfa541"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.155422 31995 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 18:58:42.157838 32015 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.159140 31995 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:42.159277 31995 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "16d6ab2428014b829cad08c3a3919e6b"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "528f2d22e637f0d0fc171660e0d6d367"
server_key_iv: "d81de71ec1db232dc56bf02d5ecfa541"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.159421 31995 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:42.181216 31995 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:42.184608 31995 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:42.184828 31995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:42.185452 31995 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:42.186309 31995 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:42.186380 31995 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.186457 31995 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:42.186502 31995 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.195626 31995 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:38877
I20260810 18:58:42.195675 32184 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:38877 every 8 connection(s)
I20260810 18:58:42.196581 31995 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:42.200133 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31995
I20260810 18:58:42.200269 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:42.200570 28238 external_mini_cluster.cc:1428] Setting key 78a50708cc1ddafad63d3c4acafcf94d
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[31836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:42.212285 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:42.198469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:39647 (local address 127.27.147.190:41279)
0810 18:58:42.198626 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:42.198629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:42.199482 (+ 853us) server_negotiation.cc:407] Connection header received
0810 18:58:42.201190 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:42.201193 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:42.201251 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:42.201360 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:42.202719 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.203315 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.204813 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.205213 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.205350 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:42.208230 (+ 2880us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:42.208233 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:42.208235 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:42.208273 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:42.209810 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:42.210412 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:42.210414 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:42.210415 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:42.210493 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:42.210973 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:42.210978 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:42.210982 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:42.211181 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:42.211266 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:42.211983 (+ 717us) server_negotiation.cc:299] Negotiation successful
0810 18:58:42.212140 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:58:42.213150 32189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:42.198857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41279 (local address 127.27.147.129:39647)
0810 18:58:42.199332 (+ 475us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:42.199375 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:42.200965 (+ 1590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:42.201543 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:42.201550 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:42.201892 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:42.202559 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:42.202583 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.203443 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.203450 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:42.204684 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:42.204694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.205393 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.205396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:42.205532 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:42.206147 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:42.206164 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:42.208039 (+ 1875us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:42.209918 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:42.209924 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:42.209939 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:42.210310 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:42.210593 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:42.210598 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:42.210600 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:42.210798 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:42.211306 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:42.211313 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:42.211651 (+ 338us) client_negotiation.cc:769] Sending connection context
0810 18:58:42.211871 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 18:58:42.212237 (+ 366us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":113,"threads_started":1}
I20260810 18:58:42.214392 32185 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41279
I20260810 18:58:42.214622 32185 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 18:58:42.215242 32185 heartbeater.cc:511] Master 127.27.147.190:41279 requested a full tablet report, sending...
I20260810 18:58:42.216887 31898 ts_manager.cc:194] Registered new tserver with Master: 16d6ab2428014b829cad08c3a3919e6b (127.27.147.129:38877)
I20260810 18:58:42.218595 31898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:39647
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:42.232131 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41279
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:42.346493 32196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:42.346731 32196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:42.346773 32196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:42.350831 32196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:42.350888 32196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:42.350945 32196 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:42.356046 32196 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:42.356984 32196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:42.358140 32196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:42.368191 32206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:42.368257 32208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:42.368196 32205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:42.368402 32196 server_base.cc:1034] running on GCE node
I20260810 18:58:42.368933 32196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:42.369609 32196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:42.370805 32196 hybrid_clock.cc:648] HybridClock initialized: now 1786388322370790 us; error 25 us; skew 500 ppm
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[31836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:42.373785 32196 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:42.374696 32196 webserver.cc:460] Webserver started at http://127.27.147.130:36403/ using document root <none> and password file <none>
I20260810 18:58:42.375322 32196 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:42.375419 32196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:42.375656 32196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:42.377677 32196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "401e10734e54470cab2248337af3306e"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "4a7b8fcff3b54f09ee287a8701ca31d9"
server_key_iv: "2231d5e7221d1b00eaf5ed8c1f701619"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.378244 32196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "401e10734e54470cab2248337af3306e"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "4a7b8fcff3b54f09ee287a8701ca31d9"
server_key_iv: "2231d5e7221d1b00eaf5ed8c1f701619"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.382254 32196 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:58:42.384836 32217 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.386193 32196 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:42.386307 32196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "401e10734e54470cab2248337af3306e"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "4a7b8fcff3b54f09ee287a8701ca31d9"
server_key_iv: "2231d5e7221d1b00eaf5ed8c1f701619"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.386384 32196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:42.407873 32196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:42.410950 32196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:42.411150 32196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:42.411803 32196 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:42.412714 32196 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:42.412762 32196 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.412827 32196 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:42.412864 32196 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.421582 32196 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:33631
I20260810 18:58:42.421651 32389 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:33631 every 8 connection(s)
I20260810 18:58:42.422492 32196 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:42.429322 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 32196
I20260810 18:58:42.429419 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:42.429715 28238 external_mini_cluster.cc:1428] Setting key 6051a5e5d99f6523c40250ad2be01bf3
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[31836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:42.436846 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:42.424361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:36763 (local address 127.27.147.190:41279)
0810 18:58:42.424501 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:42.424505 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:42.425264 (+ 759us) server_negotiation.cc:407] Connection header received
0810 18:58:42.426527 (+ 1263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:42.426530 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:42.426579 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:42.426652 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:42.428189 (+ 1537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.428754 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.430174 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.430419 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.430531 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:42.432987 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:42.432991 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:42.432994 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:42.433026 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:42.434453 (+ 1427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:42.435059 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:42.435066 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:42.435070 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:42.435170 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:42.435578 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:42.435584 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:42.435589 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:42.435857 (+ 268us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:42.435965 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:42.436466 (+ 501us) server_negotiation.cc:299] Negotiation successful
0810 18:58:42.436618 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:58:42.437592 32395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:42.424671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41279 (local address 127.27.147.130:36763)
0810 18:58:42.425108 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:42.425150 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:42.426315 (+ 1165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:42.426870 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:42.426883 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:42.427230 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:42.428013 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:42.428028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.428893 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.428896 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:42.430038 (+ 1142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:42.430048 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.430584 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.430589 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:42.430751 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:42.431398 (+ 647us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:42.431415 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:42.432824 (+ 1409us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:42.434596 (+ 1772us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:42.434600 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:42.434612 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:42.434955 (+ 343us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:42.435296 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:42.435299 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:42.435300 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:42.435462 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:42.435971 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:42.435976 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:42.436201 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 18:58:42.436371 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 18:58:42.436695 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":117,"threads_started":1}
I20260810 18:58:42.438745 32392 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41279
I20260810 18:58:42.439033 32392 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:42.439553 32392 heartbeater.cc:511] Master 127.27.147.190:41279 requested a full tablet report, sending...
I20260810 18:58:42.440637 31898 ts_manager.cc:194] Registered new tserver with Master: 401e10734e54470cab2248337af3306e (127.27.147.130:33631)
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:58:42.441226 31898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:36763
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:42.459998 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41279
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:37703
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:42.573112 32399 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:42.573395 32399 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:42.573484 32399 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:42.577443 32399 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:42.577526 32399 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:42.577618 32399 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:42.583086 32399 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37703
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41279
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32399
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:42.584199 32399 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:42.585220 32399 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:42.594961 32409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:42.594970 32415 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:42.594985 32410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:42.595347 32399 server_base.cc:1034] running on GCE node
I20260810 18:58:42.595778 32399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:42.596455 32399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:42.597631 32399 hybrid_clock.cc:648] HybridClock initialized: now 1786388322597599 us; error 44 us; skew 500 ppm
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[31836](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:42.600418 32399 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:42.601264 32399 webserver.cc:460] Webserver started at http://127.27.147.131:36031/ using document root <none> and password file <none>
I20260810 18:58:42.601854 32399 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:42.601949 32399 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:42.602183 32399 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:42.604281 32399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "033b7f39c4ac4b71855ea408e89f9b20"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "936a62df1a7fbad61bf870db1cb30fff"
server_key_iv: "840234dbde7a03a421dfc4f6b22ac04d"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.604848 32399 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "033b7f39c4ac4b71855ea408e89f9b20"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "936a62df1a7fbad61bf870db1cb30fff"
server_key_iv: "840234dbde7a03a421dfc4f6b22ac04d"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.608668 32399 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:58:42.611318 32427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.612650 32399 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:58:42.612798 32399 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "033b7f39c4ac4b71855ea408e89f9b20"
format_stamp: "Formatted at 2026-08-10 18:58:42 on dist-test-slave-znh6"
server_key: "936a62df1a7fbad61bf870db1cb30fff"
server_key_iv: "840234dbde7a03a421dfc4f6b22ac04d"
server_key_version: "encryptionkey@0"
I20260810 18:58:42.612916 32399 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:42.623677 32399 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:42.626652 32399 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:42.626909 32399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:42.627504 32399 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:42.628342 32399 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:42.628415 32399 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.628499 32399 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:42.628541 32399 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:42.637790 32399 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:45769
I20260810 18:58:42.637817 32598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:45769 every 8 connection(s)
I20260810 18:58:42.638674 32399 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:42.646910 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 32399
I20260810 18:58:42.647081 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:42.647456 28238 external_mini_cluster.cc:1428] Setting key b94048f5305590fc31d25af1369925d5
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[31836](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388322, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:42.652911 31953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:42.640616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:59131 (local address 127.27.147.190:41279)
0810 18:58:42.640762 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:42.640767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:42.641584 (+ 817us) server_negotiation.cc:407] Connection header received
0810 18:58:42.642973 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:42.642977 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:42.643039 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:42.643151 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:42.644308 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.644860 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.646036 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.646276 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.646389 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:42.649300 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:42.649303 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:42.649305 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:42.649330 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:42.650798 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:42.651385 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:42.651389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:42.651391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:42.651485 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:42.651833 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:42.651838 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:42.651839 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:42.652026 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:42.652139 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:42.652622 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 18:58:42.652743 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:58:42.653667 32601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:42.640974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41279 (local address 127.27.147.131:59131)
0810 18:58:42.641421 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:42.641475 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:42.642688 (+ 1213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:42.643337 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:42.643343 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:42.643689 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:42.644158 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:42.644171 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.645005 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.645013 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:42.645900 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:42.645909 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:42.646451 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:42.646456 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:42.646664 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:42.647441 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:42.647462 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:42.649120 (+ 1658us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:42.650989 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:42.650995 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:42.651009 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:42.651272 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:42.651582 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:42.651586 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:42.651588 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:42.651731 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:42.652138 (+ 407us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:42.652146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:42.652385 (+ 239us) client_negotiation.cc:769] Sending connection context
0810 18:58:42.652577 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 18:58:42.652883 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":122,"threads_started":1}
I20260810 18:58:42.654724 32599 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41279
I20260810 18:58:42.654997 32599 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:42.655488 32599 heartbeater.cc:511] Master 127.27.147.190:41279 requested a full tablet report, sending...
I20260810 18:58:42.656522 31898 ts_manager.cc:194] Registered new tserver with Master: 033b7f39c4ac4b71855ea408e89f9b20 (127.27.147.131:45769)
I20260810 18:58:42.657083 31898 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:59131
I20260810 18:58:42.661453 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:58:42.677763 31953 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:42.676382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34712 (local address 127.27.147.190:41279)
0810 18:58:42.676520 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:42.676524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:42.676709 (+ 185us) server_negotiation.cc:407] Connection header received
0810 18:58:42.676817 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:42.676819 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:42.676863 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:42.676940 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:42.677601 (+ 661us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34712: BlockingRecv error: recv got EOF from 127.0.0.1:34712 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:58:42.678988 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31995
I20260810 18:58:42.696898 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 32196
I20260810 18:58:42.710234 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 32399
I20260810 18:58:42.720707 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31854
2026-08-10T18:58:42Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (2407 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[32615](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[32615](info): set up 2 sockets
Aug 10 18:58:42 dist-test-slave-znh6 krb5kdc[32615](info): commencing operation
krb5kdc: starting...
W20260810 18:58:43.968997 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:44 dist-test-slave-znh6 krb5kdc[32615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:44.096312 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37601
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:41021
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:44.213163 32630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:44.213408 32630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:44.213491 32630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:44.217502 32630 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:44.217557 32630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:44.217576 32630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:44.217633 32630 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:44.217666 32630 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:44.222896 32630 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:41021
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:44.223912 32630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:44.224941 32630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:44.233634 32644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:44.233682 32638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:44.233724 32630 server_base.cc:1034] running on GCE node
W20260810 18:58:44.233634 32641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:44.234422 32630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:44.235514 32630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:44.236714 32630 hybrid_clock.cc:648] HybridClock initialized: now 1786388324236684 us; error 39 us; skew 500 ppm
Aug 10 18:58:44 dist-test-slave-znh6 krb5kdc[32615](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388324, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:44.239512 32630 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:44.240379 32630 webserver.cc:460] Webserver started at http://127.27.147.190:38531/ using document root <none> and password file <none>
I20260810 18:58:44.240960 32630 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:44.241055 32630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:44.241295 32630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:44.243528 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "bd041bc20675453ca85e337268161b43"
format_stamp: "Formatted at 2026-08-10 18:58:44 on dist-test-slave-znh6"
server_key: "ab8f629ac38c5b99464094d6e4f60f46"
server_key_iv: "dbff468f8295d9e2993dd41fd7b6fe33"
server_key_version: "encryptionkey@0"
I20260810 18:58:44.244122 32630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "bd041bc20675453ca85e337268161b43"
format_stamp: "Formatted at 2026-08-10 18:58:44 on dist-test-slave-znh6"
server_key: "ab8f629ac38c5b99464094d6e4f60f46"
server_key_iv: "dbff468f8295d9e2993dd41fd7b6fe33"
server_key_version: "encryptionkey@0"
I20260810 18:58:44.247925 32630 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:58:44.250630 32656 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:44.251977 32630 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:58:44.252127 32630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "bd041bc20675453ca85e337268161b43"
format_stamp: "Formatted at 2026-08-10 18:58:44 on dist-test-slave-znh6"
server_key: "ab8f629ac38c5b99464094d6e4f60f46"
server_key_iv: "dbff468f8295d9e2993dd41fd7b6fe33"
server_key_version: "encryptionkey@0"
I20260810 18:58:44.252260 32630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:44.268361 32630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:44.271374 32630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:44.271585 32630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:44.279250 32630 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37601
I20260810 18:58:44.279335 32719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37601 every 8 connection(s)
I20260810 18:58:44.280241 32630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:44.283090 32721 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:44.283491 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 32630
I20260810 18:58:44.283619 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:44.283910 28238 external_mini_cluster.cc:1428] Setting key 81a548b0e9a671b36c6abefccedc256c
I20260810 18:58:44.290290 32721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43: Bootstrap starting.
I20260810 18:58:44.292760 32721 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43: Neither blocks nor log segments found. Creating new log.
Aug 10 18:58:44 dist-test-slave-znh6 krb5kdc[32615](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388324, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:44.293687 32721 log.cc:824] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:44.295938 32721 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43: No bootstrap required, opened a new log
I20260810 18:58:44.298704 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:44.285190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56864 (local address 127.27.147.190:37601)
0810 18:58:44.285613 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:44.285632 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:44.285677 (+ 45us) server_negotiation.cc:407] Connection header received
0810 18:58:44.286495 (+ 818us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:44.286519 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:44.286984 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:44.287352 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:44.288034 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:44.289577 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:44.290488 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:44.291047 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:44.291226 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:44.294017 (+ 2791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:44.294023 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:44.294035 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:44.294062 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:44.295769 (+ 1707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:44.296255 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:44.296260 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:44.296264 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:44.296374 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:44.296685 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:44.296687 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:44.296688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:44.297104 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:44.297244 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:44.297635 (+ 391us) server_negotiation.cc:299] Negotiation successful
0810 18:58:44.297983 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":269,"thread_start_us":114,"threads_started":1}
I20260810 18:58:44.300007 32721 raft_consensus.cc:348] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd041bc20675453ca85e337268161b43" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37601 } }
I20260810 18:58:44.300280 32721 raft_consensus.cc:374] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:44.300330 32721 raft_consensus.cc:729] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd041bc20675453ca85e337268161b43, State: Initialized, Role: FOLLOWER
I20260810 18:58:44.300786 32721 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd041bc20675453ca85e337268161b43" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37601 } }
I20260810 18:58:44.300925 32721 raft_consensus.cc:388] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:44.301039 32721 raft_consensus.cc:482] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:44.301175 32721 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:44.301780 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:58:44.302223 32721 raft_consensus.cc:504] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd041bc20675453ca85e337268161b43" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37601 } }
I20260810 18:58:44.302557 32721 leader_election.cc:302] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bd041bc20675453ca85e337268161b43; no voters:
I20260810 18:58:44.302933 32721 leader_election.cc:288] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:44.303015 32727 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:44.303220 32727 raft_consensus.cc:686] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [term 1 LEADER]: Becoming Leader. State: Replica: bd041bc20675453ca85e337268161b43, State: Running, Role: LEADER
I20260810 18:58:44.303611 32727 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd041bc20675453ca85e337268161b43" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37601 } }
I20260810 18:58:44.304272 32721 sys_catalog.cc:564] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:44.304863 32727 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd041bc20675453ca85e337268161b43. Latest consensus state: current_term: 1 leader_uuid: "bd041bc20675453ca85e337268161b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd041bc20675453ca85e337268161b43" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37601 } } }
I20260810 18:58:44.304981 32727 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:44.305332 32728 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bd041bc20675453ca85e337268161b43" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bd041bc20675453ca85e337268161b43" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37601 } } }
I20260810 18:58:44.305516 32728 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43 [sys.catalog]: This master's current role is: LEADER
W20260810 18:58:44.309370 32750 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:58:44.309471 32750 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:58:44.309561 32732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:44.310552 32732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:44.316145 32732 catalog_manager.cc:1347] Generated new cluster ID: 10b1bcd6b6c64d9ebdf77f543c3a8708
I20260810 18:58:44.316205 32732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:44.323843 32724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:44.317982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56868 (local address 127.27.147.190:37601)
0810 18:58:44.318087 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:44.318092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:44.318429 (+ 337us) server_negotiation.cc:407] Connection header received
0810 18:58:44.318562 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:44.318565 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:44.318624 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:44.318714 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:44.319318 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:44.320175 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:44.321326 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:44.321686 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:44.321809 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:44.322734 (+ 925us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:44.322739 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:44.322743 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:44.323350 (+ 607us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:44.323389 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:44.323410 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:44.323507 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:44.323619 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 18:58:44.323685 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:58:44.328732 32732 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:44.329967 32732 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:44.345839 32732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bd041bc20675453ca85e337268161b43: Generated new TSK 0
I20260810 18:58:44.346760 32732 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:58:44.360636 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 32630
2026-08-10T18:58:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1644 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:44 dist-test-slave-znh6 krb5kdc[307](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:44 dist-test-slave-znh6 krb5kdc[307](info): set up 2 sockets
Aug 10 18:58:44 dist-test-slave-znh6 krb5kdc[307](info): commencing operation
krb5kdc: starting...
W20260810 18:58:45.615402 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.223s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:45 dist-test-slave-znh6 krb5kdc[307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:45.737836 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37727
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:45.849529 325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:45.849774 325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:45.849812 325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:45.853818 325 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:45.853869 325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:45.853888 325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:45.853904 325 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:45.853974 325 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:45.859287 325 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:45.860313 325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:45.861322 325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:45.870028 332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:45.870042 335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:45.870059 337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:45.870410 325 server_base.cc:1034] running on GCE node
I20260810 18:58:45.871002 325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:45.872026 325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:45.873220 325 hybrid_clock.cc:648] HybridClock initialized: now 1786388325873208 us; error 25 us; skew 500 ppm
Aug 10 18:58:45 dist-test-slave-znh6 krb5kdc[307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388325, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:45.875979 325 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:45.876914 325 webserver.cc:460] Webserver started at http://127.27.147.190:37605/ using document root <none> and password file <none>
I20260810 18:58:45.877485 325 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:45.877578 325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:45.877813 325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:45.879889 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "f48d29bfb0da4ecf9d9610fb3da11a56"
format_stamp: "Formatted at 2026-08-10 18:58:45 on dist-test-slave-znh6"
server_key: "4264e65cf3bcc794ce8c03255d7795db"
server_key_iv: "14268fb88542141c16adff01e53ee0fa"
server_key_version: "encryptionkey@0"
I20260810 18:58:45.880442 325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "f48d29bfb0da4ecf9d9610fb3da11a56"
format_stamp: "Formatted at 2026-08-10 18:58:45 on dist-test-slave-znh6"
server_key: "4264e65cf3bcc794ce8c03255d7795db"
server_key_iv: "14268fb88542141c16adff01e53ee0fa"
server_key_version: "encryptionkey@0"
I20260810 18:58:45.884145 325 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:58:45.886451 350 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:45.887681 325 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:45.887837 325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "f48d29bfb0da4ecf9d9610fb3da11a56"
format_stamp: "Formatted at 2026-08-10 18:58:45 on dist-test-slave-znh6"
server_key: "4264e65cf3bcc794ce8c03255d7795db"
server_key_iv: "14268fb88542141c16adff01e53ee0fa"
server_key_version: "encryptionkey@0"
I20260810 18:58:45.887946 325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:45.899752 325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:45.902283 325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:45.902484 325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:45.909442 325 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37727
I20260810 18:58:45.909724 423 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37727 every 8 connection(s)
I20260810 18:58:45.910373 325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:45.913094 425 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:45.914023 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 325
I20260810 18:58:45.914158 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:45.914471 28238 external_mini_cluster.cc:1428] Setting key 684ecc76d996edbee4a6290f775dbff1
I20260810 18:58:45.919626 425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56: Bootstrap starting.
I20260810 18:58:45.922284 425 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:45.923177 425 log.cc:824] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:58:45 dist-test-slave-znh6 krb5kdc[307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:45.925529 425 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56: No bootstrap required, opened a new log
I20260810 18:58:45.929199 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:45.915912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55722 (local address 127.27.147.190:37727)
0810 18:58:45.916338 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:45.916353 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:45.916387 (+ 34us) server_negotiation.cc:407] Connection header received
0810 18:58:45.916999 (+ 612us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:45.917015 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:45.917318 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:45.917592 (+ 274us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:45.918492 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:45.919513 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:45.920624 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:45.921063 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:45.921176 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:45.924079 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:45.924088 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:45.924107 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:45.924136 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:45.926037 (+ 1901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:45.926492 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:45.926498 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:45.926504 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:45.926638 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:45.927008 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:45.927013 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:45.927016 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:45.927514 (+ 498us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:45.927729 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:45.927964 (+ 235us) server_negotiation.cc:299] Negotiation successful
0810 18:58:45.928282 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":143,"threads_started":1}
I20260810 18:58:45.930083 425 raft_consensus.cc:348] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37727 } }
I20260810 18:58:45.930346 425 raft_consensus.cc:374] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:45.930433 425 raft_consensus.cc:729] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f48d29bfb0da4ecf9d9610fb3da11a56, State: Initialized, Role: FOLLOWER
I20260810 18:58:45.930904 425 consensus_queue.cc:260] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37727 } }
I20260810 18:58:45.931027 425 raft_consensus.cc:388] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:45.931111 425 raft_consensus.cc:482] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:45.931205 425 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:45.932189 425 raft_consensus.cc:504] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37727 } }
I20260810 18:58:45.932528 425 leader_election.cc:302] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f48d29bfb0da4ecf9d9610fb3da11a56; no voters:
I20260810 18:58:45.932819 425 leader_election.cc:288] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:45.932946 434 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:45.933183 434 raft_consensus.cc:686] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [term 1 LEADER]: Becoming Leader. State: Replica: f48d29bfb0da4ecf9d9610fb3da11a56, State: Running, Role: LEADER
I20260810 18:58:45.933516 434 consensus_queue.cc:237] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37727 } }
I20260810 18:58:45.934039 425 sys_catalog.cc:564] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:45.935122 435 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37727 } } }
I20260810 18:58:45.935253 435 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:45.935535 436 sys_catalog.cc:455] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f48d29bfb0da4ecf9d9610fb3da11a56. Latest consensus state: current_term: 1 leader_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f48d29bfb0da4ecf9d9610fb3da11a56" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37727 } } }
I20260810 18:58:45.935647 445 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:45.935691 436 sys_catalog.cc:458] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:45.938649 445 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:45.943660 445 catalog_manager.cc:1347] Generated new cluster ID: dd904598ecbe4feda4cedd835c06f9f4
I20260810 18:58:45.943725 445 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:45.961453 445 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:45.962677 445 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:45.976579 445 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f48d29bfb0da4ecf9d9610fb3da11a56: Generated new TSK 0
I20260810 18:58:45.977411 445 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:46.017256 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37727
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:46.129876 469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:46.130126 469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:46.130172 469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:46.134291 469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:46.134438 469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:46.134548 469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:46.139782 469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.469
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:46.140816 469 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:46.141835 469 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:46.151696 476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:46.151732 481 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:46.151705 478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:46.151872 469 server_base.cc:1034] running on GCE node
I20260810 18:58:46.152396 469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:46.153086 469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:46.154275 469 hybrid_clock.cc:648] HybridClock initialized: now 1786388326154262 us; error 25 us; skew 500 ppm
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:46.156996 469 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:46.157837 469 webserver.cc:460] Webserver started at http://127.27.147.129:42157/ using document root <none> and password file <none>
I20260810 18:58:46.158361 469 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:46.158473 469 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:46.158699 469 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:46.160778 469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "42a3801c198a41d88db3e9d304853f34"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "fbd9ea3f2b8e4d0a93b819d4cdc0cacf"
server_key_iv: "4f00495218c1a326f98fe08793fbd20b"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.161352 469 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "42a3801c198a41d88db3e9d304853f34"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "fbd9ea3f2b8e4d0a93b819d4cdc0cacf"
server_key_iv: "4f00495218c1a326f98fe08793fbd20b"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.165149 469 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:58:46.167555 498 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.168805 469 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:46.168943 469 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "42a3801c198a41d88db3e9d304853f34"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "fbd9ea3f2b8e4d0a93b819d4cdc0cacf"
server_key_iv: "4f00495218c1a326f98fe08793fbd20b"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.169062 469 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:46.181775 469 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:46.184864 469 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:46.185071 469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:46.185668 469 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:46.186527 469 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:46.186602 469 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.186676 469 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:46.186720 469 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.196087 469 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:45409
I20260810 18:58:46.196115 655 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:45409 every 8 connection(s)
I20260810 18:58:46.196990 469 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:46.203698 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 469
I20260810 18:58:46.203862 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:46.204164 28238 external_mini_cluster.cc:1428] Setting key d1f3c01501a46720b99233fee7eae0e5
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:46.211617 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:46.198876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:55613 (local address 127.27.147.190:37727)
0810 18:58:46.199018 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:46.199022 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:46.199838 (+ 816us) server_negotiation.cc:407] Connection header received
0810 18:58:46.201375 (+ 1537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:46.201379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:46.201440 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:46.201544 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:46.202691 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.203322 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.204629 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.204950 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.205073 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:46.207763 (+ 2690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:46.207766 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:46.207768 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:46.207790 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:46.209272 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.209896 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.209899 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.209900 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.209989 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.210386 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.210389 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.210390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.210556 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:46.210633 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:46.211325 (+ 692us) server_negotiation.cc:299] Negotiation successful
0810 18:58:46.211472 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:58:46.212760 659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:46.199229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37727 (local address 127.27.147.129:55613)
0810 18:58:46.199691 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:46.199732 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:46.201148 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:46.201724 (+ 576us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:46.201730 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:46.202075 (+ 345us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:46.202543 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:46.202555 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.203458 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.203461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:46.204498 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:46.204508 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.205109 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.205114 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:46.205332 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:46.205913 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:46.205929 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:46.207590 (+ 1661us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:46.209397 (+ 1807us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:46.209405 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:46.209423 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:46.209787 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:46.210091 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:46.210097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:46.210102 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:46.210290 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:46.210663 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:46.210672 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:46.211031 (+ 359us) client_negotiation.cc:769] Sending connection context
0810 18:58:46.211275 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 18:58:46.211642 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":120,"threads_started":1}
I20260810 18:58:46.214025 656 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37727
I20260810 18:58:46.214293 656 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:46.214920 656 heartbeater.cc:511] Master 127.27.147.190:37727 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:46.216506 375 ts_manager.cc:194] Registered new tserver with Master: 42a3801c198a41d88db3e9d304853f34 (127.27.147.129:45409)
I20260810 18:58:46.217664 375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:55613
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:46.234298 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37727
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:46.345484 664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:46.345726 664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:46.345765 664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:46.349722 664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:46.349769 664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:46.349867 664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:46.354874 664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:46.355862 664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:46.356845 664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:46.366609 673 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:46.366627 670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:46.366631 669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:46.366796 664 server_base.cc:1034] running on GCE node
I20260810 18:58:46.367347 664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:46.367951 664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:46.369114 664 hybrid_clock.cc:648] HybridClock initialized: now 1786388326369100 us; error 25 us; skew 500 ppm
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:46.371932 664 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:46.372789 664 webserver.cc:460] Webserver started at http://127.27.147.130:37097/ using document root <none> and password file <none>
I20260810 18:58:46.373375 664 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:46.373440 664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:46.373672 664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:46.376188 664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "cacb96c087ce4d76993f827eeb4e97b6"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "5c7468dac8b0cc7d117215a0c83ea1e3"
server_key_iv: "d3dfde6d5196d29cf388635dd2e7adad"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.376749 664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "cacb96c087ce4d76993f827eeb4e97b6"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "5c7468dac8b0cc7d117215a0c83ea1e3"
server_key_iv: "d3dfde6d5196d29cf388635dd2e7adad"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.380422 664 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 18:58:46.382715 682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.383927 664 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:46.384049 664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "cacb96c087ce4d76993f827eeb4e97b6"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "5c7468dac8b0cc7d117215a0c83ea1e3"
server_key_iv: "d3dfde6d5196d29cf388635dd2e7adad"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.384147 664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:46.405125 664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:46.408592 664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:46.408800 664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:46.409396 664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:46.410231 664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:46.410302 664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.410367 664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:46.410399 664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.419898 664 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:39675
I20260810 18:58:46.419921 852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:39675 every 8 connection(s)
I20260810 18:58:46.420794 664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:46.430862 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 664
I20260810 18:58:46.430990 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:46.431281 28238 external_mini_cluster.cc:1428] Setting key 765e42f0e29ae6573b583f8ae2148bc9
I20260810 18:58:46.434721 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:46.422545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:35865 (local address 127.27.147.190:37727)
0810 18:58:46.422709 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:46.422714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:46.423558 (+ 844us) server_negotiation.cc:407] Connection header received
0810 18:58:46.424885 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:46.424888 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:46.424934 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:46.425016 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:46.426109 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.426628 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.428359 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.428675 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.428759 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:46.431151 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:46.431156 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:46.431158 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:46.431191 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:46.432635 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.433154 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.433162 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.433163 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.433235 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.433584 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.433587 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.433589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.433766 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:46.433875 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:46.434423 (+ 548us) server_negotiation.cc:299] Negotiation successful
0810 18:58:46.434551 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:58:46.435699 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:46.422969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37727 (local address 127.27.147.130:35865)
0810 18:58:46.423411 (+ 442us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:46.423452 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:46.424693 (+ 1241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:46.425177 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:46.425184 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:46.425518 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:46.425986 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:46.425999 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.426731 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.426733 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:46.428232 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:46.428242 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.428806 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.428808 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:46.428949 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:46.429459 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:46.429475 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:46.430972 (+ 1497us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:46.432767 (+ 1795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:46.432773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:46.432786 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:46.433042 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:46.433329 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:46.433333 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:46.433335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:46.433479 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:46.433873 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:46.433880 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:46.434113 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 18:58:46.434363 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 18:58:46.434668 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260810 18:58:46.436872 853 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37727
I20260810 18:58:46.437114 853 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:46.437686 853 heartbeater.cc:511] Master 127.27.147.190:37727 requested a full tablet report, sending...
I20260810 18:58:46.438889 375 ts_manager.cc:194] Registered new tserver with Master: cacb96c087ce4d76993f827eeb4e97b6 (127.27.147.130:39675)
I20260810 18:58:46.439481 375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:35865
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:46.460362 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37727
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:36377
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:46.571303 859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:46.571578 859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:46.571669 859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:46.575778 859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:46.575857 859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:46.575945 859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:46.581169 859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36377
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37727
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.859
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:46.582173 859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:46.583232 859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:46.592983 868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:46.593006 870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:46.592990 866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:46.593039 859 server_base.cc:1034] running on GCE node
I20260810 18:58:46.593665 859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:46.594317 859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:46.595477 859 hybrid_clock.cc:648] HybridClock initialized: now 1786388326595447 us; error 40 us; skew 500 ppm
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:46.598248 859 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:46.599148 859 webserver.cc:460] Webserver started at http://127.27.147.131:34617/ using document root <none> and password file <none>
I20260810 18:58:46.599690 859 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:46.599753 859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:46.600006 859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:46.602001 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "1310ae7cd3314212b4629d8d7221b665"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "24656d232de58324ab6868503bd67098"
server_key_iv: "07c2cceb929c06761d58c724cb906cbf"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.602564 859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "1310ae7cd3314212b4629d8d7221b665"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "24656d232de58324ab6868503bd67098"
server_key_iv: "07c2cceb929c06761d58c724cb906cbf"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.606174 859 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:46.608525 879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.609687 859 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:46.609807 859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "1310ae7cd3314212b4629d8d7221b665"
format_stamp: "Formatted at 2026-08-10 18:58:46 on dist-test-slave-znh6"
server_key: "24656d232de58324ab6868503bd67098"
server_key_iv: "07c2cceb929c06761d58c724cb906cbf"
server_key_version: "encryptionkey@0"
I20260810 18:58:46.609910 859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:46.629637 859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:46.632845 859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:46.633059 859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:46.633720 859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:46.634532 859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:46.634580 859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.634649 859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:46.634690 859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:46.643201 859 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:40873
I20260810 18:58:46.643277 1056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:40873 every 8 connection(s)
I20260810 18:58:46.644125 859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:46.646903 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 859
I20260810 18:58:46.647032 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:46.647367 28238 external_mini_cluster.cc:1428] Setting key 0e4f470907cfa90e8142427a11fc5ab2
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:46.657310 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:46.645861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:57507 (local address 127.27.147.190:37727)
0810 18:58:46.646012 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:46.646015 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:46.646883 (+ 868us) server_negotiation.cc:407] Connection header received
0810 18:58:46.648613 (+ 1730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:46.648619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:46.648678 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:46.648769 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:46.650014 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.650542 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.651653 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.651867 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.651954 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:46.654147 (+ 2193us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:46.654150 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:46.654152 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:46.654175 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:46.655669 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.656103 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.656105 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.656106 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.656176 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.656456 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.656458 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.656459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.656620 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:46.656696 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:46.657101 (+ 405us) server_negotiation.cc:299] Negotiation successful
0810 18:58:46.657191 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 18:58:46.658025 1059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:46.646214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37727 (local address 127.27.147.131:57507)
0810 18:58:46.646681 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:46.646739 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:46.648358 (+ 1619us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:46.648960 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:46.648967 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:46.649291 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:46.649889 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:46.649901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.650644 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.650647 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:46.651550 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:46.651560 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.651998 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.652000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:46.652129 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:46.652621 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:46.652637 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:46.654006 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:46.655768 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:46.655773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:46.655783 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:46.656017 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:46.656254 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:46.656256 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:46.656258 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:46.656375 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:46.656706 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:46.656710 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:46.656910 (+ 200us) client_negotiation.cc:769] Sending connection context
0810 18:58:46.657052 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:58:46.657270 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":107,"threads_started":1}
I20260810 18:58:46.659067 1057 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37727
I20260810 18:58:46.659265 1057 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:46.659677 1057 heartbeater.cc:511] Master 127.27.147.190:37727 requested a full tablet report, sending...
I20260810 18:58:46.660728 375 ts_manager.cc:194] Registered new tserver with Master: 1310ae7cd3314212b4629d8d7221b665 (127.27.147.131:40873)
I20260810 18:58:46.661291 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:58:46.661469 375 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:57507
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[307](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388326, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:46.689240 431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:46.680474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55732 (local address 127.27.147.190:37727)
0810 18:58:46.680651 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:46.680656 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:46.680804 (+ 148us) server_negotiation.cc:407] Connection header received
0810 18:58:46.680979 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:46.680983 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:46.681032 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:46.681117 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:46.681829 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.682355 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.683300 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:46.683530 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:46.683645 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:46.685990 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:46.685994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:46.685996 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:46.686026 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:46.687513 (+ 1487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.687954 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.687958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.687960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.688040 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:46.688334 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:46.688338 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:46.688339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:46.688551 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:46.688697 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:46.688955 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 18:58:46.689076 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 18:58:46.689630 375 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:55732
I20260810 18:58:46.690420 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 469
I20260810 18:58:46.707617 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 664
I20260810 18:58:46.726714 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 859
I20260810 18:58:46.734791 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 325
2026-08-10T18:58:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (2372 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[1074](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[1074](info): set up 2 sockets
Aug 10 18:58:46 dist-test-slave-znh6 krb5kdc[1074](info): commencing operation
krb5kdc: starting...
W20260810 18:58:47.975467 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.211s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:48.101745 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:40907
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:40907
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:48.212874 1102 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:48.213114 1102 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:48.213153 1102 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:48.217063 1102 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:48.217114 1102 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:48.217133 1102 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:48.217149 1102 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:48.217163 1102 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:48.222332 1102 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:40907
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:40907
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1102
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:48.223462 1102 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:48.224495 1102 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:48.233206 1109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:48.233227 1112 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:48.233333 1102 server_base.cc:1034] running on GCE node
W20260810 18:58:48.233201 1108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:48.234082 1102 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:48.235289 1102 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:48.236479 1102 hybrid_clock.cc:648] HybridClock initialized: now 1786388328236453 us; error 40 us; skew 500 ppm
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:48.245163 1102 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:48.246058 1102 webserver.cc:460] Webserver started at http://127.27.147.190:33973/ using document root <none> and password file <none>
I20260810 18:58:48.246592 1102 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:48.246650 1102 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:48.246871 1102 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:48.248967 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "0c68651680364c75ae67db3b7e1bffaf"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "e0598d20734135eddeff2041320c98b2"
server_key_iv: "dd4df341b31837f0d3b3c03604eafcae"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.249485 1102 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "0c68651680364c75ae67db3b7e1bffaf"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "e0598d20734135eddeff2041320c98b2"
server_key_iv: "dd4df341b31837f0d3b3c03604eafcae"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.253175 1102 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:58:48.255524 1124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.256712 1102 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:48.256824 1102 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "0c68651680364c75ae67db3b7e1bffaf"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "e0598d20734135eddeff2041320c98b2"
server_key_iv: "dd4df341b31837f0d3b3c03604eafcae"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.256901 1102 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:48.280090 1102 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:48.283322 1102 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:48.283476 1102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:48.290772 1102 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:40907
I20260810 18:58:48.290798 1196 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:40907 every 8 connection(s)
I20260810 18:58:48.291795 1102 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:48.294549 1197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:48.298074 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 1102
I20260810 18:58:48.298172 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:48.298461 28238 external_mini_cluster.cc:1428] Setting key ca73a70a596b1fc7f4d50a6b1826b298
I20260810 18:58:48.300637 1197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf: Bootstrap starting.
I20260810 18:58:48.303234 1197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:48.304088 1197 log.cc:824] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:48.306224 1197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf: No bootstrap required, opened a new log
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:48.310781 1197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68651680364c75ae67db3b7e1bffaf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40907 } }
I20260810 18:58:48.311084 1197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:48.311162 1197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c68651680364c75ae67db3b7e1bffaf, State: Initialized, Role: FOLLOWER
I20260810 18:58:48.311641 1197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68651680364c75ae67db3b7e1bffaf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40907 } }
I20260810 18:58:48.311810 1197 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:48.311944 1197 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:48.312069 1197 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:48.313087 1197 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68651680364c75ae67db3b7e1bffaf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40907 } }
I20260810 18:58:48.313128 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:48.299847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37258 (local address 127.27.147.190:40907)
0810 18:58:48.300250 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:48.300277 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:48.300341 (+ 64us) server_negotiation.cc:407] Connection header received
0810 18:58:48.300991 (+ 650us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:48.301011 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:48.301328 (+ 317us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:48.301593 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:48.302533 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.303651 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.304771 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.305255 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.305380 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:48.308083 (+ 2703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:48.308090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:48.308106 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:48.308137 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:48.310071 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:48.310475 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:48.310481 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:48.310487 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:48.310622 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:48.310941 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:48.310944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:48.310946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:48.311438 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:48.311637 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:48.311847 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 18:58:48.312161 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":117,"threads_started":1}
I20260810 18:58:48.313469 1197 leader_election.cc:302] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c68651680364c75ae67db3b7e1bffaf; no voters:
I20260810 18:58:48.313795 1197 leader_election.cc:288] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:48.313918 1209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:48.314148 1209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [term 1 LEADER]: Becoming Leader. State: Replica: 0c68651680364c75ae67db3b7e1bffaf, State: Running, Role: LEADER
I20260810 18:58:48.314495 1209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68651680364c75ae67db3b7e1bffaf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40907 } }
I20260810 18:58:48.315006 1197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:48.316282 1210 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c68651680364c75ae67db3b7e1bffaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68651680364c75ae67db3b7e1bffaf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40907 } } }
I20260810 18:58:48.316411 1210 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:48.316781 1223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:48.316759 1211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c68651680364c75ae67db3b7e1bffaf. Latest consensus state: current_term: 1 leader_uuid: "0c68651680364c75ae67db3b7e1bffaf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c68651680364c75ae67db3b7e1bffaf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40907 } } }
I20260810 18:58:48.316910 1211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:48.319963 1223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:48.325599 1223 catalog_manager.cc:1347] Generated new cluster ID: 689c87e53ecf4bf697365e9f9945fd17
I20260810 18:58:48.325661 1223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:48.340412 1223 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:48.341358 1223 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:48.347258 1223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c68651680364c75ae67db3b7e1bffaf: Generated new TSK 0
I20260810 18:58:48.347934 1223 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:48.385462 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:40907
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:48.503922 1242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:48.504165 1242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:48.504207 1242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:48.508234 1242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:48.508285 1242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:48.508391 1242 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:48.513579 1242 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:40907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:48.514599 1242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:48.515766 1242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:48.525599 1252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:48.525707 1248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:48.525619 1249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:48.525866 1242 server_base.cc:1034] running on GCE node
I20260810 18:58:48.526309 1242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:48.527009 1242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:48.528198 1242 hybrid_clock.cc:648] HybridClock initialized: now 1786388328528187 us; error 29 us; skew 500 ppm
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:48.530946 1242 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:48.531769 1242 webserver.cc:460] Webserver started at http://127.27.147.129:41259/ using document root <none> and password file <none>
I20260810 18:58:48.532326 1242 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:48.532420 1242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:48.532660 1242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:48.534694 1242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "bf0c94402db1498e84f9fb237e04e105"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "534f4e02e46a910fed0c48852a905beb"
server_key_iv: "18e6d332626039208ef1e5cfac65f15b"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.535383 1242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "bf0c94402db1498e84f9fb237e04e105"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "534f4e02e46a910fed0c48852a905beb"
server_key_iv: "18e6d332626039208ef1e5cfac65f15b"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.539016 1242 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:58:48.541268 1269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.542475 1242 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:48.542605 1242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "bf0c94402db1498e84f9fb237e04e105"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "534f4e02e46a910fed0c48852a905beb"
server_key_iv: "18e6d332626039208ef1e5cfac65f15b"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.542711 1242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:48.556892 1242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:48.559509 1242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:48.559734 1242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:48.560328 1242 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:48.561185 1242 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:48.561235 1242 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.561303 1242 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:48.561335 1242 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.570482 1242 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:36021
I20260810 18:58:48.570519 1439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:36021 every 8 connection(s)
I20260810 18:58:48.571403 1242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:48.572341 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 1242
I20260810 18:58:48.572486 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:48.572768 28238 external_mini_cluster.cc:1428] Setting key 79656428ce40bb25c72662af00ba71c1
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:48.585827 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:48.573471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:48859 (local address 127.27.147.190:40907)
0810 18:58:48.573612 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:48.573616 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:48.574341 (+ 725us) server_negotiation.cc:407] Connection header received
0810 18:58:48.575825 (+ 1484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:48.575831 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:48.575900 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:48.576004 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:48.577130 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.578001 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.579218 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.579476 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.579597 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:48.582111 (+ 2514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:48.582115 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:48.582118 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:48.582153 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:48.583744 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:48.584283 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:48.584287 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:48.584289 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:48.584372 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:48.584720 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:48.584725 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:48.584727 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:48.584914 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:48.585031 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:48.585524 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 18:58:48.585651 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 18:58:48.586771 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:48.573693 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:40907 (local address 127.27.147.129:48859)
0810 18:58:48.574187 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:48.574238 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:48.575615 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:48.576158 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:48.576164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:48.576520 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:48.576978 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:48.576991 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.578125 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.578128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:48.579092 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:48.579101 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.579641 (+ 540us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.579645 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:48.579854 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:48.580451 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:48.580468 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:48.581945 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:48.583869 (+ 1924us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:48.583875 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:48.583891 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:48.584170 (+ 279us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:48.584476 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:48.584479 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:48.584481 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:48.584620 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:48.585031 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:48.585039 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:48.585283 (+ 244us) client_negotiation.cc:769] Sending connection context
0810 18:58:48.585482 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 18:58:48.585796 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":143,"threads_started":1}
I20260810 18:58:48.588009 1440 heartbeater.cc:347] Connected to a master server at 127.27.147.190:40907
I20260810 18:58:48.588294 1440 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:48.588935 1440 heartbeater.cc:511] Master 127.27.147.190:40907 requested a full tablet report, sending...
I20260810 18:58:48.590564 1149 ts_manager.cc:194] Registered new tserver with Master: bf0c94402db1498e84f9fb237e04e105 (127.27.147.129:36021)
I20260810 18:58:48.591769 1149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:48859
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:48.603317 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:40907
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:48.715229 1445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:48.715458 1445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:48.715529 1445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:48.719695 1445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:48.719745 1445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:48.719841 1445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:48.724861 1445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:40907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:48.725795 1445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:48.726754 1445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:58:48.736310 1445 server_base.cc:1034] running on GCE node
W20260810 18:58:48.736250 1454 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:48.736258 1451 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:48.736254 1458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:48.736863 1445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:48.737463 1445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:48.738651 1445 hybrid_clock.cc:648] HybridClock initialized: now 1786388328738640 us; error 24 us; skew 500 ppm
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:48.741370 1445 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:48.742147 1445 webserver.cc:460] Webserver started at http://127.27.147.130:43321/ using document root <none> and password file <none>
I20260810 18:58:48.742643 1445 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:48.742700 1445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:48.742934 1445 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:48.744839 1445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "41e5eb55ff6d445188d142332ef67fd9"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "b41d49421a8a6df322d01a4b50615248"
server_key_iv: "05fa76c01f06df9a9a2a99eb60d70c7a"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.745352 1445 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "41e5eb55ff6d445188d142332ef67fd9"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "b41d49421a8a6df322d01a4b50615248"
server_key_iv: "05fa76c01f06df9a9a2a99eb60d70c7a"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.749094 1445 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:58:48.751387 1470 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.752596 1445 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:48.752699 1445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "41e5eb55ff6d445188d142332ef67fd9"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "b41d49421a8a6df322d01a4b50615248"
server_key_iv: "05fa76c01f06df9a9a2a99eb60d70c7a"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.752780 1445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:48.770149 1445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:48.772791 1445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:48.772941 1445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:48.773427 1445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:48.774174 1445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:48.774220 1445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.774258 1445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:48.774279 1445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.783113 1445 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:46231
I20260810 18:58:48.783143 1644 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:46231 every 8 connection(s)
I20260810 18:58:48.783975 1445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:48.789723 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 1445
I20260810 18:58:48.789841 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:48.790159 28238 external_mini_cluster.cc:1428] Setting key 9e37636830a047d908fa30617a4b7862
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:48.797860 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:48.785708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:35803 (local address 127.27.147.190:40907)
0810 18:58:48.785853 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:48.785859 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:48.786659 (+ 800us) server_negotiation.cc:407] Connection header received
0810 18:58:48.788024 (+ 1365us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:48.788027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:48.788075 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:48.788150 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:48.789226 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.789967 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.791277 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.791606 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.791706 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:48.794437 (+ 2731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:48.794441 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:48.794446 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:48.794473 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:48.795847 (+ 1374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:48.796354 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:48.796359 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:48.796361 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:48.796442 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:48.796779 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:48.796784 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:48.796786 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:48.796972 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:48.797082 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:48.797556 (+ 474us) server_negotiation.cc:299] Negotiation successful
0810 18:58:48.797692 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:58:48.798681 1649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:48.786066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:40907 (local address 127.27.147.130:35803)
0810 18:58:48.786506 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:48.786546 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:48.787835 (+ 1289us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:48.788306 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:48.788313 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:48.788656 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:48.789103 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:48.789115 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.790090 (+ 975us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.790093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:48.791152 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:48.791162 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:48.791770 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:48.791775 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:48.791972 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:48.792545 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:48.792571 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:48.794275 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:48.795974 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:48.795979 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:48.795993 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:48.796246 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:48.796538 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:48.796542 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:48.796544 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:48.796680 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:48.797081 (+ 401us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:48.797087 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:48.797321 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 18:58:48.797512 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 18:58:48.797837 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":111,"threads_started":1}
I20260810 18:58:48.799834 1645 heartbeater.cc:347] Connected to a master server at 127.27.147.190:40907
I20260810 18:58:48.800099 1645 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:48.800565 1645 heartbeater.cc:511] Master 127.27.147.190:40907 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:58:48.801662 1149 ts_manager.cc:194] Registered new tserver with Master: 41e5eb55ff6d445188d142332ef67fd9 (127.27.147.130:46231)
I20260810 18:58:48.802263 1149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:35803
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:48.818161 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:40907
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40223
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:48.933308 1653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:48.933569 1653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:48.933614 1653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:48.937706 1653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:48.937764 1653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:48.937862 1653 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:48.943100 1653 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40223
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:40907
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:48.944208 1653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:48.945261 1653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:48.955040 1661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:48.955053 1663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:48.955082 1665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:48.955274 1653 server_base.cc:1034] running on GCE node
I20260810 18:58:48.955693 1653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:48.956296 1653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:48.957487 1653 hybrid_clock.cc:648] HybridClock initialized: now 1786388328957467 us; error 32 us; skew 500 ppm
Aug 10 18:58:48 dist-test-slave-znh6 krb5kdc[1074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:48.960318 1653 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:48.961171 1653 webserver.cc:460] Webserver started at http://127.27.147.131:35545/ using document root <none> and password file <none>
I20260810 18:58:48.961678 1653 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:48.961736 1653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:48.961895 1653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:48.963986 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "06503d5545c34732b3627c7d83858a94"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "4f1feb26cca44689bd60fce325e07a7a"
server_key_iv: "5bcf349382ec255dcd5d3e6507eddaaf"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.964547 1653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "06503d5545c34732b3627c7d83858a94"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "4f1feb26cca44689bd60fce325e07a7a"
server_key_iv: "5bcf349382ec255dcd5d3e6507eddaaf"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.968106 1653 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:58:48.970378 1678 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:48.971602 1653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:48.971741 1653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "06503d5545c34732b3627c7d83858a94"
format_stamp: "Formatted at 2026-08-10 18:58:48 on dist-test-slave-znh6"
server_key: "4f1feb26cca44689bd60fce325e07a7a"
server_key_iv: "5bcf349382ec255dcd5d3e6507eddaaf"
server_key_version: "encryptionkey@0"
I20260810 18:58:48.971851 1653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:49.000861 1653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:49.004287 1653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:49.004488 1653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:49.005095 1653 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:49.005966 1653 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:49.006040 1653 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:49.006111 1653 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:49.006165 1653 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:49.015215 1653 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:42321
I20260810 18:58:49.015239 1856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:42321 every 8 connection(s)
I20260810 18:58:49.016145 1653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:49.024653 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 1653
I20260810 18:58:49.024811 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:49.025183 28238 external_mini_cluster.cc:1428] Setting key 6535c10ce68e6ca3974ad6c90fca5050
Aug 10 18:58:49 dist-test-slave-znh6 krb5kdc[1074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388328, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:49.029733 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:49.018001 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:38617 (local address 127.27.147.190:40907)
0810 18:58:49.018143 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:49.018147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:49.018914 (+ 767us) server_negotiation.cc:407] Connection header received
0810 18:58:49.020300 (+ 1386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:49.020304 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:49.020367 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:49.020474 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:49.021609 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.022156 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.023392 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.023619 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.023729 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:49.026476 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:49.026481 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:49.026484 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:49.026512 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:49.028015 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:49.028464 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:49.028466 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:49.028468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:49.028544 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:49.028830 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:49.028832 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:49.028833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:49.029011 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:49.029088 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:49.029520 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 18:58:49.029617 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:58:49.030551 1860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:49.018294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:40907 (local address 127.27.147.131:38617)
0810 18:58:49.018741 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:49.018782 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:49.020086 (+ 1304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:49.020657 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:49.020663 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:49.021005 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:49.021464 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:49.021477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.022283 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.022292 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:49.023241 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:49.023263 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.023774 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.023778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:49.023941 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:49.024469 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:49.024485 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:49.026296 (+ 1811us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:49.028117 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:49.028121 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:49.028133 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:49.028370 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:49.028623 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:49.028625 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:49.028626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:49.028748 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:49.029098 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:49.029102 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:49.029315 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 18:58:49.029475 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 18:58:49.029724 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":122,"threads_started":1}
I20260810 18:58:49.031741 1858 heartbeater.cc:347] Connected to a master server at 127.27.147.190:40907
I20260810 18:58:49.031971 1858 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:49.032438 1858 heartbeater.cc:511] Master 127.27.147.190:40907 requested a full tablet report, sending...
I20260810 18:58:49.033428 1149 ts_manager.cc:194] Registered new tserver with Master: 06503d5545c34732b3627c7d83858a94 (127.27.147.131:42321)
I20260810 18:58:49.033985 1149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:38617
I20260810 18:58:49.039147 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 18:58:49 dist-test-slave-znh6 krb5kdc[1074](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:58:49 dist-test-slave-znh6 krb5kdc[1074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:49.067169 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:49.057326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51980 (local address 127.27.147.129:36021)
0810 18:58:49.057613 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:49.057630 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:49.057657 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:58:49.057746 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:49.057760 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:49.057983 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:49.058146 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:49.058871 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.059655 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.060547 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.060907 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.061031 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:49.063549 (+ 2518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:49.063559 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:49.063571 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:49.063593 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:49.065238 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:49.065766 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:49.065772 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:49.065773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:49.065852 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:49.066143 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:49.066147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:49.066149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:49.066386 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:49.066620 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:49.066807 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 18:58:49.066996 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
Aug 10 18:58:49 dist-test-slave-znh6 krb5kdc[1074](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:49.077797 1207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:49.069815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37274 (local address 127.27.147.190:40907)
0810 18:58:49.069967 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:49.069971 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:49.069999 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:58:49.070103 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:49.070105 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:49.070145 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:49.070211 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:49.070744 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.071296 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.072005 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:49.072211 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:49.072284 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:49.074693 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:49.074697 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:49.074698 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:49.074721 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:49.076232 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:49.076701 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:49.076703 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:49.076705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:49.076781 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:49.077002 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:49.077004 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:49.077005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:49.077197 (+ 192us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:49.077325 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:49.077485 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 18:58:49.077575 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 18:58:49.078235 1149 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37274
I20260810 18:58:49.079002 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 1242
I20260810 18:58:49.093699 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 1445
I20260810 18:58:49.105611 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 1653
I20260810 18:58:49.118690 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 1102
2026-08-10T18:58:49Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (2381 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:49 dist-test-slave-znh6 krb5kdc[1873](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:49 dist-test-slave-znh6 krb5kdc[1873](info): set up 2 sockets
Aug 10 18:58:49 dist-test-slave-znh6 krb5kdc[1873](info): commencing operation
krb5kdc: starting...
W20260810 18:58:50.371203 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.225s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:50 dist-test-slave-znh6 krb5kdc[1873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:50.501545 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:44073
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:44073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:50.613940 1897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:50.614177 1897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:50.614218 1897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:50.618165 1897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:50.618216 1897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:50.618234 1897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:50.618274 1897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:50.618311 1897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:50.623382 1897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:44073
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:44073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:50.624348 1897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:50.625353 1897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:50.633504 1903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:50.633504 1905 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:50.633513 1911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:50.633757 1897 server_base.cc:1034] running on GCE node
I20260810 18:58:50.634456 1897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:50.635607 1897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:50.636804 1897 hybrid_clock.cc:648] HybridClock initialized: now 1786388330636775 us; error 40 us; skew 500 ppm
Aug 10 18:58:50 dist-test-slave-znh6 krb5kdc[1873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:50.639582 1897 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:50.640290 1897 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:50.640393 1897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:50.640627 1897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:50.642611 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "65ed4bf4fab04b02911147ee727e1983"
format_stamp: "Formatted at 2026-08-10 18:58:50 on dist-test-slave-znh6"
server_key: "1bfe7df219e50bfb02ce8c987c75d4dc"
server_key_iv: "321b9d0303b63b2bce24e128c9952fa8"
server_key_version: "encryptionkey@0"
I20260810 18:58:50.643229 1897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "65ed4bf4fab04b02911147ee727e1983"
format_stamp: "Formatted at 2026-08-10 18:58:50 on dist-test-slave-znh6"
server_key: "1bfe7df219e50bfb02ce8c987c75d4dc"
server_key_iv: "321b9d0303b63b2bce24e128c9952fa8"
server_key_version: "encryptionkey@0"
I20260810 18:58:50.646768 1897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 18:58:50.649076 1925 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:50.650254 1897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:50.650413 1897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "65ed4bf4fab04b02911147ee727e1983"
format_stamp: "Formatted at 2026-08-10 18:58:50 on dist-test-slave-znh6"
server_key: "1bfe7df219e50bfb02ce8c987c75d4dc"
server_key_iv: "321b9d0303b63b2bce24e128c9952fa8"
server_key_version: "encryptionkey@0"
I20260810 18:58:50.650528 1897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:50.681715 1897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:50.684989 1897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:50.685207 1897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:50.692544 1897 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:44073
I20260810 18:58:50.692566 2011 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:44073 every 8 connection(s)
I20260810 18:58:50.693197 1897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:50.695956 2012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:50.697957 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 1897
I20260810 18:58:50.698069 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:50.698345 28238 external_mini_cluster.cc:1428] Setting key 31d457d833cf21d128e4a6b2565ffef6
I20260810 18:58:50.702633 2012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983: Bootstrap starting.
I20260810 18:58:50.704870 2012 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:50.705714 2012 log.cc:824] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983: Log is configured to *not* fsync() on all Append() calls
Aug 10 18:58:50 dist-test-slave-znh6 krb5kdc[1873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:50.708050 2012 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983: No bootstrap required, opened a new log
I20260810 18:58:50.712335 2012 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ed4bf4fab04b02911147ee727e1983" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 44073 } }
I20260810 18:58:50.712551 2012 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:50.712622 2012 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65ed4bf4fab04b02911147ee727e1983, State: Initialized, Role: FOLLOWER
I20260810 18:58:50.712721 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:50.699852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51652 (local address 127.27.147.190:44073)
0810 18:58:50.700302 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:50.700325 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:50.700361 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:58:50.700995 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:50.701011 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:50.701321 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:50.701671 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:50.702481 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:50.703574 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:50.704441 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:50.704910 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:50.705108 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:50.708093 (+ 2985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:50.708098 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:50.708133 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:50.708174 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:50.709783 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:50.710340 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:50.710345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:50.710350 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:50.710461 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:50.710752 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:50.710754 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:50.710755 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:50.711177 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:50.711332 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:50.711624 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 18:58:50.711936 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":124,"threads_started":1}
I20260810 18:58:50.713128 2012 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ed4bf4fab04b02911147ee727e1983" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 44073 } }
I20260810 18:58:50.713286 2012 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:50.713375 2012 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:50.713501 2012 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:50.714571 2012 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ed4bf4fab04b02911147ee727e1983" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 44073 } }
I20260810 18:58:50.714944 2012 leader_election.cc:302] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65ed4bf4fab04b02911147ee727e1983; no voters:
I20260810 18:58:50.715305 2012 leader_election.cc:288] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:50.715365 2019 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:50.715618 2019 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [term 1 LEADER]: Becoming Leader. State: Replica: 65ed4bf4fab04b02911147ee727e1983, State: Running, Role: LEADER
I20260810 18:58:50.715965 2019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ed4bf4fab04b02911147ee727e1983" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 44073 } }
I20260810 18:58:50.716516 2012 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:50.717684 2020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65ed4bf4fab04b02911147ee727e1983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ed4bf4fab04b02911147ee727e1983" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 44073 } } }
I20260810 18:58:50.717828 2020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:50.718218 2034 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:50.718173 2022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65ed4bf4fab04b02911147ee727e1983. Latest consensus state: current_term: 1 leader_uuid: "65ed4bf4fab04b02911147ee727e1983" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65ed4bf4fab04b02911147ee727e1983" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 44073 } } }
I20260810 18:58:50.718436 2022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:50.721145 2034 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:50.728312 2034 catalog_manager.cc:1347] Generated new cluster ID: 0115485584b041ecbd61a14985ec1f8a
I20260810 18:58:50.728394 2034 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:50.736783 2034 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:50.737665 2034 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:50.746897 2034 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983: Generated new TSK 0
I20260810 18:58:50.747560 2034 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:50.777805 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:44073
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:50.891057 2057 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:50.891295 2057 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:50.891337 2057 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:50.895414 2057 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:50.895478 2057 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:50.895536 2057 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:50.900624 2057 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:44073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2057
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:50.901635 2057 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:50.902729 2057 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:50.912235 2070 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:50.912231 2068 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:50.912246 2067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:50.912513 2057 server_base.cc:1034] running on GCE node
I20260810 18:58:50.912961 2057 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:50.913633 2057 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:50.914803 2057 hybrid_clock.cc:648] HybridClock initialized: now 1786388330914784 us; error 31 us; skew 500 ppm
Aug 10 18:58:50 dist-test-slave-znh6 krb5kdc[1873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:50.917798 2057 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:50.918581 2057 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:50.918644 2057 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:50.918807 2057 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:50.920795 2057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "a7722d91d5dd4f6bbebfe361dfa5e969"
format_stamp: "Formatted at 2026-08-10 18:58:50 on dist-test-slave-znh6"
server_key: "af49fb88612272cb97e92262d2c72fa3"
server_key_iv: "574b460ddfdd36f2e689d18e62da1721"
server_key_version: "encryptionkey@0"
I20260810 18:58:50.921298 2057 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "a7722d91d5dd4f6bbebfe361dfa5e969"
format_stamp: "Formatted at 2026-08-10 18:58:50 on dist-test-slave-znh6"
server_key: "af49fb88612272cb97e92262d2c72fa3"
server_key_iv: "574b460ddfdd36f2e689d18e62da1721"
server_key_version: "encryptionkey@0"
I20260810 18:58:50.924952 2057 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.004s
I20260810 18:58:50.927264 2078 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:50.928431 2057 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:50.928535 2057 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "a7722d91d5dd4f6bbebfe361dfa5e969"
format_stamp: "Formatted at 2026-08-10 18:58:50 on dist-test-slave-znh6"
server_key: "af49fb88612272cb97e92262d2c72fa3"
server_key_iv: "574b460ddfdd36f2e689d18e62da1721"
server_key_version: "encryptionkey@0"
I20260810 18:58:50.928615 2057 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:50.945348 2057 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:50.948648 2057 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:50.948799 2057 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:50.949298 2057 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:50.950094 2057 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:50.950143 2057 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:50.950182 2057 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:50.950202 2057 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:50.959434 2057 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:32979
I20260810 18:58:50.959527 2245 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:32979 every 8 connection(s)
I20260810 18:58:50.960064 2057 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:50.964186 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 2057
I20260810 18:58:50.964322 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:50.964619 28238 external_mini_cluster.cc:1428] Setting key 8563d1a24b0858e1bdc30848f8ed0589
Aug 10 18:58:50 dist-test-slave-znh6 krb5kdc[1873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388330, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:50.974897 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:50.961803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:43041 (local address 127.27.147.190:44073)
0810 18:58:50.961936 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:50.961940 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:50.962769 (+ 829us) server_negotiation.cc:407] Connection header received
0810 18:58:50.964146 (+ 1377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:50.964150 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:50.964213 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:50.964318 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:50.965819 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:50.966622 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:50.967895 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:50.968190 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:50.968311 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:50.971292 (+ 2981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:50.971296 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:50.971299 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:50.971326 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:50.972792 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:50.973333 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:50.973337 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:50.973338 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:50.973412 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:50.973769 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:50.973772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:50.973774 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:50.973953 (+ 179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:50.974063 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:50.974568 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 18:58:50.974697 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:50.975785 2249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:50.962179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:44073 (local address 127.27.147.129:43041)
0810 18:58:50.962620 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:50.962662 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:50.963923 (+ 1261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:50.964551 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:50.964559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:50.964998 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:50.965674 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:50.965692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:50.966758 (+ 1066us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:50.966761 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:50.967755 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:50.967769 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:50.968515 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:50.968522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:50.968751 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:50.969392 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:50.969413 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:50.971126 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:50.972928 (+ 1802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:50.972934 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:50.972949 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:50.973213 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:50.973515 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:50.973519 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:50.973521 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:50.973664 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:50.974063 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:50.974070 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:50.974321 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 18:58:50.974512 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 18:58:50.974832 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":111,"threads_started":1}
I20260810 18:58:50.977134 2247 heartbeater.cc:347] Connected to a master server at 127.27.147.190:44073
I20260810 18:58:50.977465 2247 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 18:58:50.978017 2247 heartbeater.cc:511] Master 127.27.147.190:44073 requested a full tablet report, sending...
I20260810 18:58:50.979771 1955 ts_manager.cc:194] Registered new tserver with Master: a7722d91d5dd4f6bbebfe361dfa5e969 (127.27.147.129:32979)
I20260810 18:58:50.981437 1955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:43041
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:50.994351 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:44073
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:51.105301 2253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:51.105538 2253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:51.105580 2253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:51.109408 2253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:51.109455 2253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:51.109510 2253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:51.114418 2253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:44073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:51.115334 2253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:51.116228 2253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:51.125813 2266 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:51.125824 2263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:51.125874 2253 server_base.cc:1034] running on GCE node
W20260810 18:58:51.125814 2264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:51.126448 2253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:51.127106 2253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:51.128260 2253 hybrid_clock.cc:648] HybridClock initialized: now 1786388331128234 us; error 38 us; skew 500 ppm
Aug 10 18:58:51 dist-test-slave-znh6 krb5kdc[1873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:51.131048 2253 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:51.131848 2253 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:51.131914 2253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:51.132154 2253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:51.134192 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "bcf7c84cebed4912addc286131b49a4c"
format_stamp: "Formatted at 2026-08-10 18:58:51 on dist-test-slave-znh6"
server_key: "197acdc9577d5e2e9e347ac7b3c3d79d"
server_key_iv: "26e87dbb882c1e30e294054029f19c50"
server_key_version: "encryptionkey@0"
I20260810 18:58:51.134735 2253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "bcf7c84cebed4912addc286131b49a4c"
format_stamp: "Formatted at 2026-08-10 18:58:51 on dist-test-slave-znh6"
server_key: "197acdc9577d5e2e9e347ac7b3c3d79d"
server_key_iv: "26e87dbb882c1e30e294054029f19c50"
server_key_version: "encryptionkey@0"
I20260810 18:58:51.138192 2253 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:51.140542 2276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:51.141705 2253 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 18:58:51.141820 2253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "bcf7c84cebed4912addc286131b49a4c"
format_stamp: "Formatted at 2026-08-10 18:58:51 on dist-test-slave-znh6"
server_key: "197acdc9577d5e2e9e347ac7b3c3d79d"
server_key_iv: "26e87dbb882c1e30e294054029f19c50"
server_key_version: "encryptionkey@0"
I20260810 18:58:51.141927 2253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:51.163894 2253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:51.166538 2253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:51.166738 2253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:51.167337 2253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:51.168169 2253 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:51.168216 2253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:51.168284 2253 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:51.168322 2253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:51.177186 2253 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:42465
I20260810 18:58:51.177204 2457 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:42465 every 8 connection(s)
I20260810 18:58:51.177762 2253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:51.181291 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 2253
I20260810 18:58:51.181388 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:51.181716 28238 external_mini_cluster.cc:1428] Setting key 3350e7e37d577404b41e50ed99e9fdb7
Aug 10 18:58:51 dist-test-slave-znh6 krb5kdc[1873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:51.191845 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.179499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:51379 (local address 127.27.147.190:44073)
0810 18:58:51.179646 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.179650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.180388 (+ 738us) server_negotiation.cc:407] Connection header received
0810 18:58:51.181830 (+ 1442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.181833 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.181896 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.181982 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:51.183347 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.183877 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.185005 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.185239 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.185352 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.188109 (+ 2757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:51.188112 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:51.188114 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:51.188136 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:51.189584 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:51.190224 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:51.190226 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:51.190228 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:51.190298 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:51.190706 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:51.190708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:51.190709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:51.190929 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:51.191015 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.191629 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.191727 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:58:51.193029 2465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.179799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:44073 (local address 127.27.147.130:51379)
0810 18:58:51.180231 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.180272 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:51.181613 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:51.182164 (+ 551us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:51.182172 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:51.182598 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:51.183189 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.183203 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.184005 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.184014 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.184868 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.184878 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.185396 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.185401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.185577 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.186165 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:51.186182 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:51.187927 (+ 1745us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:51.189708 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:51.189717 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:51.189734 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:51.190121 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:51.190397 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:51.190403 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:51.190407 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:51.190605 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:51.191043 (+ 438us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:51.191049 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:51.191362 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 18:58:51.191598 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 18:58:51.191939 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":113,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:58:51.194260 2460 heartbeater.cc:347] Connected to a master server at 127.27.147.190:44073
I20260810 18:58:51.194510 2460 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:51.195016 2460 heartbeater.cc:511] Master 127.27.147.190:44073 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
I20260810 18:58:51.196203 1955 ts_manager.cc:194] Registered new tserver with Master: bcf7c84cebed4912addc286131b49a4c (127.27.147.130:42465)
I20260810 18:58:51.196763 1955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:51379
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:51.211455 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:44073
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40225
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:51.328155 2471 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:51.328388 2471 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:51.328466 2471 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:51.332638 2471 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:51.332693 2471 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:51.332782 2471 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:51.338081 2471 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40225
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:44073
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2471
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:51.339185 2471 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:51.340184 2471 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:51.349762 2476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:51.349763 2477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:51.349892 2471 server_base.cc:1034] running on GCE node
W20260810 18:58:51.350140 2479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:51.350617 2471 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:51.351274 2471 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:51.352478 2471 hybrid_clock.cc:648] HybridClock initialized: now 1786388331352440 us; error 51 us; skew 500 ppm
Aug 10 18:58:51 dist-test-slave-znh6 krb5kdc[1873](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:51.355808 2471 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:51.356568 2471 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:51.356637 2471 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:51.356858 2471 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:51.358990 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "3f1b7436a8604e01a7bd040faa2d702b"
format_stamp: "Formatted at 2026-08-10 18:58:51 on dist-test-slave-znh6"
server_key: "5a4e02a2241f31b402a2bbbc00b12615"
server_key_iv: "8e523647d8847633a5afce038a559779"
server_key_version: "encryptionkey@0"
I20260810 18:58:51.359582 2471 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "3f1b7436a8604e01a7bd040faa2d702b"
format_stamp: "Formatted at 2026-08-10 18:58:51 on dist-test-slave-znh6"
server_key: "5a4e02a2241f31b402a2bbbc00b12615"
server_key_iv: "8e523647d8847633a5afce038a559779"
server_key_version: "encryptionkey@0"
I20260810 18:58:51.363291 2471 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 18:58:51.365639 2487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:51.366988 2471 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:51.367110 2471 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "3f1b7436a8604e01a7bd040faa2d702b"
format_stamp: "Formatted at 2026-08-10 18:58:51 on dist-test-slave-znh6"
server_key: "5a4e02a2241f31b402a2bbbc00b12615"
server_key_iv: "8e523647d8847633a5afce038a559779"
server_key_version: "encryptionkey@0"
I20260810 18:58:51.367209 2471 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:51.380354 2471 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:51.383653 2471 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:51.383854 2471 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:51.384433 2471 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:51.385255 2471 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:51.385303 2471 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:51.385367 2471 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:51.385406 2471 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:51.394766 2471 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:37217
I20260810 18:58:51.394788 2663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:37217 every 8 connection(s)
I20260810 18:58:51.395471 2471 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:51.398097 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 2471
I20260810 18:58:51.398313 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:51.398636 28238 external_mini_cluster.cc:1428] Setting key 706428880e351b9e288891962a9b0c3f
Aug 10 18:58:51 dist-test-slave-znh6 krb5kdc[1873](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388331, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:51.409713 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.397251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:58957 (local address 127.27.147.190:44073)
0810 18:58:51.397449 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.397453 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.398302 (+ 849us) server_negotiation.cc:407] Connection header received
0810 18:58:51.400152 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.400157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.400211 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.400293 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:51.401718 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.402244 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.403677 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.403890 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.403974 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.406357 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:51.406361 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:51.406363 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:51.406386 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:51.407863 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:51.408360 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:51.408363 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:51.408364 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:51.408435 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:51.408737 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:51.408739 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:51.408740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:51.408902 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:51.408979 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.409488 (+ 509us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.409589 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 18:58:51.410544 2670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.397606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:44073 (local address 127.27.147.131:58957)
0810 18:58:51.398091 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.398169 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:51.399937 (+ 1768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:51.400486 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:51.400495 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:51.401012 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:51.401567 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.401582 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.402373 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.402376 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.403543 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.403554 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.404032 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.404035 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.404187 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.404709 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:51.404725 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:51.406188 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:51.407972 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:51.407999 (+ 27us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:51.408016 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:51.408272 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:51.408523 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:51.408525 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:51.408527 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:51.408656 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:51.408990 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:51.408994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:51.409218 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 18:58:51.409418 (+ 200us) client_negotiation.cc:240] Negotiation successful
0810 18:58:51.409678 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":160,"threads_started":1}
I20260810 18:58:51.411906 2666 heartbeater.cc:347] Connected to a master server at 127.27.147.190:44073
I20260810 18:58:51.412194 2666 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:51.412655 2666 heartbeater.cc:511] Master 127.27.147.190:44073 requested a full tablet report, sending...
I20260810 18:58:51.413719 1955 ts_manager.cc:194] Registered new tserver with Master: 3f1b7436a8604e01a7bd040faa2d702b (127.27.147.131:37217)
I20260810 18:58:51.414364 1955 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:58957
I20260810 18:58:51.424538 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:58:51.434099 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.426493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51666 (local address 127.27.147.190:44073)
0810 18:58:51.426652 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.426657 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.426874 (+ 217us) server_negotiation.cc:407] Connection header received
0810 18:58:51.427027 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.427034 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.427079 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.427161 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:51.427839 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.428347 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.429187 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.429411 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.429519 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.430880 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:51.430884 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:51.430887 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:51.430918 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:51.432392 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:51.432818 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:51.432827 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:51.432830 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:51.432903 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:51.433215 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:51.433220 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:51.433221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:51.433434 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:51.433583 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.433815 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.433928 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:58:51.436976 1955 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51666:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:51.440999 1955 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:51.455837 2686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.449674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:37217 (local address 127.0.0.1:35210)
0810 18:58:51.450030 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.450047 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:51.450176 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:51.451169 (+ 993us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:51.451174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:51.451195 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:51.451299 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.451303 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.452610 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.452615 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.454054 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.454104 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.455225 (+ 1121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.455229 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.455362 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.455409 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 18:58:51.455553 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 18:58:51.455677 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":26,"thread_start_us":90,"threads_started":1}
I20260810 18:58:51.455839 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.450251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35210 (local address 127.27.147.131:37217)
0810 18:58:51.450608 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.450620 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.450643 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:58:51.450736 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.450744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.450991 (+ 247us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.451178 (+ 187us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:51.451430 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.452474 (+ 1044us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.454260 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.455088 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.455226 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.455398 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.455586 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.455682 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":137,"threads_started":1}
I20260810 18:58:51.456415 2688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.450177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:42465 (local address 127.0.0.1:51144)
0810 18:58:51.450781 (+ 604us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.450793 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:51.450907 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:51.451762 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:51.451766 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:51.451787 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:51.451885 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.451891 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.453492 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.453496 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.454701 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.454712 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.456008 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.456012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.456103 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.456127 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:58:51.456215 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 18:58:51.456281 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":27,"thread_start_us":107,"threads_started":1}
I20260810 18:58:51.456678 2684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.449482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:32979 (local address 127.0.0.1:45318)
0810 18:58:51.449993 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.450041 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:51.450576 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:51.452703 (+ 2127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:51.452706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:51.452726 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:51.452824 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.452830 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.454229 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.454233 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.455176 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.455187 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.456277 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.456283 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.456389 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.456424 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 18:58:51.456489 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 18:58:51.456557 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"thread_start_us":73,"threads_started":1}
I20260810 18:58:51.456964 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.449630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45318 (local address 127.27.147.129:32979)
0810 18:58:51.451590 (+ 1960us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.451598 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.451626 (+ 28us) server_negotiation.cc:407] Connection header received
0810 18:58:51.451710 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.451715 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.451951 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.452718 (+ 767us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:51.452927 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.454042 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.455332 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.456132 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.456252 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.456629 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.456689 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.456791 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1729,"thread_start_us":101,"threads_started":1}
I20260810 18:58:51.457110 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.450589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51144 (local address 127.27.147.130:42465)
0810 18:58:51.451158 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.451170 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.451193 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:58:51.451264 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.451269 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.451572 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.451757 (+ 185us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:51.452039 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.453340 (+ 1301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.454868 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.455684 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.455804 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.455937 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.456859 (+ 922us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.456958 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":102,"threads_started":1}
I20260810 18:58:51.459756 2150 tablet_service.cc:1467] Processing CreateTablet for tablet e3f3f760035c429f8ce77069a5ca09fd (DEFAULT_TABLE table=test-table [id=b7543cacf0aa424c93d3f1b73c2c0a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:51.459671 2566 tablet_service.cc:1467] Processing CreateTablet for tablet e3f3f760035c429f8ce77069a5ca09fd (DEFAULT_TABLE table=test-table [id=b7543cacf0aa424c93d3f1b73c2c0a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:51.460059 2354 tablet_service.cc:1467] Processing CreateTablet for tablet e3f3f760035c429f8ce77069a5ca09fd (DEFAULT_TABLE table=test-table [id=b7543cacf0aa424c93d3f1b73c2c0a09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:51.460793 2150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f3f760035c429f8ce77069a5ca09fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:51.460814 2566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f3f760035c429f8ce77069a5ca09fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:51.460966 2354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3f3f760035c429f8ce77069a5ca09fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:51.466992 2691 tablet_bootstrap.cc:492] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: Bootstrap starting.
I20260810 18:58:51.468802 2694 tablet_bootstrap.cc:492] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: Bootstrap starting.
I20260810 18:58:51.469533 2691 tablet_bootstrap.cc:654] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:51.470571 2694 tablet_bootstrap.cc:654] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:51.470957 2691 log.cc:824] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:51.471223 2693 tablet_bootstrap.cc:492] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: Bootstrap starting.
I20260810 18:58:51.472146 2694 log.cc:824] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:51.473142 2691 tablet_bootstrap.cc:492] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: No bootstrap required, opened a new log
I20260810 18:58:51.473327 2691 ts_tablet_manager.cc:1397] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 18:58:51.473338 2693 tablet_bootstrap.cc:654] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:51.474150 2694 tablet_bootstrap.cc:492] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: No bootstrap required, opened a new log
I20260810 18:58:51.474323 2694 ts_tablet_manager.cc:1397] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 18:58:51.474617 2693 log.cc:824] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:51.476487 2693 tablet_bootstrap.cc:492] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: No bootstrap required, opened a new log
I20260810 18:58:51.476647 2693 ts_tablet_manager.cc:1397] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:58:51.477093 2691 raft_consensus.cc:348] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.477355 2691 raft_consensus.cc:374] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:51.477409 2691 raft_consensus.cc:729] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7722d91d5dd4f6bbebfe361dfa5e969, State: Initialized, Role: FOLLOWER
I20260810 18:58:51.477895 2691 consensus_queue.cc:260] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.478219 2694 raft_consensus.cc:348] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.478960 2694 raft_consensus.cc:374] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:51.478991 2691 ts_tablet_manager.cc:1428] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 18:58:51.479035 2694 raft_consensus.cc:729] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bcf7c84cebed4912addc286131b49a4c, State: Initialized, Role: FOLLOWER
I20260810 18:58:51.479401 2247 heartbeater.cc:503] Master 127.27.147.190:44073 was elected leader, sending a full tablet report...
I20260810 18:58:51.479517 2694 consensus_queue.cc:260] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.480196 2460 heartbeater.cc:503] Master 127.27.147.190:44073 was elected leader, sending a full tablet report...
I20260810 18:58:51.480477 2694 ts_tablet_manager.cc:1428] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:58:51.480358 2693 raft_consensus.cc:348] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.480618 2693 raft_consensus.cc:374] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:51.480688 2693 raft_consensus.cc:729] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f1b7436a8604e01a7bd040faa2d702b, State: Initialized, Role: FOLLOWER
I20260810 18:58:51.481024 2693 consensus_queue.cc:260] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.481666 2666 heartbeater.cc:503] Master 127.27.147.190:44073 was elected leader, sending a full tablet report...
I20260810 18:58:51.482913 2693 ts_tablet_manager.cc:1428] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
W20260810 18:58:51.646350 2669 tablet.cc:2369] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:58:51.678887 2461 tablet.cc:2369] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:58:51.711328 2248 tablet.cc:2369] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:58:51.800621 2702 raft_consensus.cc:482] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:51.800908 2702 raft_consensus.cc:504] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.802178 2702 leader_election.cc:288] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a7722d91d5dd4f6bbebfe361dfa5e969 (127.27.147.129:32979), 3f1b7436a8604e01a7bd040faa2d702b (127.27.147.131:37217)
I20260810 18:58:51.806540 2689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.802594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:60033 (local address 127.27.147.131:37217)
0810 18:58:51.802764 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.802769 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.802966 (+ 197us) server_negotiation.cc:407] Connection header received
0810 18:58:51.803139 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.803148 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.803225 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.803323 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:51.803639 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.804327 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.805451 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.805918 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.806051 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.806107 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.806322 (+ 215us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.806393 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 18:58:51.806540 2685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.802686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:59019 (local address 127.27.147.129:32979)
0810 18:58:51.802911 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.802917 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.803097 (+ 180us) server_negotiation.cc:407] Connection header received
0810 18:58:51.803265 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.803270 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.803322 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.803409 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:58:51.803718 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.804314 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.805430 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.805922 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.806056 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.806109 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.806343 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.806411 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 18:58:51.806555 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.802516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:32979 (local address 127.27.147.130:59019)
0810 18:58:51.803001 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.803012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:51.803148 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:51.803422 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:51.803424 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:51.803441 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:51.803604 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.803609 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.804430 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.804435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.805303 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.805312 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.806056 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.806060 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.806217 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.806235 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:58:51.806327 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:58:51.806398 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"mutex_wait_us":45,"thread_start_us":102,"threads_started":1}
I20260810 18:58:51.806577 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.802486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:37217 (local address 127.27.147.130:60033)
0810 18:58:51.802870 (+ 384us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.802883 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:51.803020 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:51.803338 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:51.803341 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:51.803358 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:58:51.803516 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.803522 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.804446 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.804450 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.805312 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:51.805321 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.806062 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.806066 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:51.806186 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.806214 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:58:51.806306 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 18:58:51.806377 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"mutex_wait_us":27,"thread_start_us":81,"threads_started":1}
I20260810 18:58:51.807427 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f3f760035c429f8ce77069a5ca09fd" candidate_uuid: "bcf7c84cebed4912addc286131b49a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1b7436a8604e01a7bd040faa2d702b" is_pre_election: true
I20260810 18:58:51.807420 2181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f3f760035c429f8ce77069a5ca09fd" candidate_uuid: "bcf7c84cebed4912addc286131b49a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" is_pre_election: true
I20260810 18:58:51.807720 2181 raft_consensus.cc:2454] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf7c84cebed4912addc286131b49a4c in term 0.
I20260810 18:58:51.807724 2594 raft_consensus.cc:2454] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bcf7c84cebed4912addc286131b49a4c in term 0.
I20260810 18:58:51.808230 2278 leader_election.cc:302] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a7722d91d5dd4f6bbebfe361dfa5e969, bcf7c84cebed4912addc286131b49a4c; no voters:
I20260810 18:58:51.808560 2702 raft_consensus.cc:2781] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:51.808646 2702 raft_consensus.cc:482] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:51.808677 2702 raft_consensus.cc:3037] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:51.860769 2702 raft_consensus.cc:504] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.861419 2702 leader_election.cc:288] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [CANDIDATE]: Term 1 election: Requested vote from peers a7722d91d5dd4f6bbebfe361dfa5e969 (127.27.147.129:32979), 3f1b7436a8604e01a7bd040faa2d702b (127.27.147.131:37217)
I20260810 18:58:51.862025 2594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f3f760035c429f8ce77069a5ca09fd" candidate_uuid: "bcf7c84cebed4912addc286131b49a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3f1b7436a8604e01a7bd040faa2d702b"
I20260810 18:58:51.862010 2181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3f3f760035c429f8ce77069a5ca09fd" candidate_uuid: "bcf7c84cebed4912addc286131b49a4c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969"
I20260810 18:58:51.862181 2594 raft_consensus.cc:3037] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:51.862181 2181 raft_consensus.cc:3037] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:51.863706 2594 raft_consensus.cc:2454] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf7c84cebed4912addc286131b49a4c in term 1.
I20260810 18:58:51.863706 2181 raft_consensus.cc:2454] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bcf7c84cebed4912addc286131b49a4c in term 1.
I20260810 18:58:51.864171 2283 leader_election.cc:302] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3f1b7436a8604e01a7bd040faa2d702b, a7722d91d5dd4f6bbebfe361dfa5e969, bcf7c84cebed4912addc286131b49a4c; no voters:
I20260810 18:58:51.864378 2702 raft_consensus.cc:2781] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:51.864642 2702 raft_consensus.cc:686] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 1 LEADER]: Becoming Leader. State: Replica: bcf7c84cebed4912addc286131b49a4c, State: Running, Role: LEADER
I20260810 18:58:51.865097 2702 consensus_queue.cc:237] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } }
I20260810 18:58:51.869076 1954 catalog_manager.cc:5495] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c reported cstate change: term changed from 0 to 1, leader changed from <none> to bcf7c84cebed4912addc286131b49a4c (127.27.147.130). New cstate: current_term: 1 leader_uuid: "bcf7c84cebed4912addc286131b49a4c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bcf7c84cebed4912addc286131b49a4c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42465 } health_report { overall_health: HEALTHY } } }
I20260810 18:58:51.924506 2690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.920763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51160 (local address 127.27.147.130:42465)
0810 18:58:51.920932 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:51.920937 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:51.920953 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:58:51.921073 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:51.921076 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:51.921141 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:51.921253 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:58:51.921522 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.922056 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.922964 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:51.923208 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:51.923324 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:51.923656 (+ 332us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:58:51.924102 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:58:51.924235 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:51.924303 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 18:58:51.924362 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:58:51.930706 2181 raft_consensus.cc:1261] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969 [term 1 FOLLOWER]: Refusing update from remote peer bcf7c84cebed4912addc286131b49a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:51.931177 2594 raft_consensus.cc:1261] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 1 FOLLOWER]: Refusing update from remote peer bcf7c84cebed4912addc286131b49a4c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:51.931519 2706 consensus_queue.cc:1035] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "a7722d91d5dd4f6bbebfe361dfa5e969" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 32979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:51.931891 2702 consensus_queue.cc:1035] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [LEADER]: Connected to new peer: Peer: permanent_uuid: "3f1b7436a8604e01a7bd040faa2d702b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 37217 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:51.939901 2714 mvcc.cc:205] Tried to move back new op lower bound from 7317046607580418048 to 7317046607323168768. Current Snapshot: MvccSnapshot[applied={T|T < 7317046607580418048}]
I20260810 18:58:51.948817 1954 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:51.948997 1954 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51666:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:51.951435 1954 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 65ed4bf4fab04b02911147ee727e1983: Sending DeleteTablet for 3 replicas of tablet e3f3f760035c429f8ce77069a5ca09fd
I20260810 18:58:51.952214 2566 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f3f760035c429f8ce77069a5ca09fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:51 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:35210
I20260810 18:58:51.952211 2150 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f3f760035c429f8ce77069a5ca09fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:51 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:45318
I20260810 18:58:51.952731 2726 tablet_replica.cc:330] T e3f3f760035c429f8ce77069a5ca09fd P a7722d91d5dd4f6bbebfe361dfa5e969: stopping tablet replica
I20260810 18:58:51.952979 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 2057
I20260810 18:58:51.952987 2354 tablet_service.cc:1514] Processing DeleteTablet for tablet e3f3f760035c429f8ce77069a5ca09fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:51 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:51144
I20260810 18:58:51.963110 2728 tablet_replica.cc:330] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: stopping tablet replica
I20260810 18:58:51.963485 2728 raft_consensus.cc:2229] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:51.963925 2728 raft_consensus.cc:2258] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:51.966197 2728 ts_tablet_manager.cc:1905] T e3f3f760035c429f8ce77069a5ca09fd P bcf7c84cebed4912addc286131b49a4c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:51.967082 2727 tablet_replica.cc:330] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: stopping tablet replica
I20260810 18:58:51.967398 2727 raft_consensus.cc:2229] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:51.967756 2727 raft_consensus.cc:2258] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:51.969337 2727 ts_tablet_manager.cc:1905] T e3f3f760035c429f8ce77069a5ca09fd P 3f1b7436a8604e01a7bd040faa2d702b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 18:58:51.976495 1930 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:32979 (error 108)
I20260810 18:58:51.976559 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 2253
I20260810 18:58:51.991338 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:51.982744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:32979 (local address 127.0.0.1:45332)
0810 18:58:51.991136 (+ 8392us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:51.991229 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":8293,"thread_start_us":304,"threads_started":1}
W20260810 18:58:51.991592 1930 catalog_manager.cc:4584] TS a7722d91d5dd4f6bbebfe361dfa5e969 (127.27.147.129:32979): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e3f3f760035c429f8ce77069a5ca09fd: Network error: Client connection negotiation failed: client connection to 127.27.147.129:32979: connect: Connection refused (error 111)
I20260810 18:58:52.044780 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.044430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:32979 (local address 127.0.0.1:45334)
0810 18:58:52.044630 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.044691 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 18:58:52.045557 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.045318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:32979 (local address 127.0.0.1:45338)
0810 18:58:52.045439 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.045497 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:32979: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 18:58:52.053767 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 2471
I20260810 18:58:52.060166 2730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.053900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:42465 (local address 127.0.0.1:51162)
0810 18:58:52.059976 (+ 6076us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.060060 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.130:42465: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5961}
I20260810 18:58:52.071633 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 1897
2026-08-10T18:58:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2951 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T18:58:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:52Z Disabled control of system clock
I20260810 18:58:52.107051 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37543
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37543
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:58:52.224050 2737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:52.224308 2737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:52.224385 2737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:52.228379 2737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:52.228432 2737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:52.228447 2737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:52.228463 2737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:52.228525 2737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:52.233500 2737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37543
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:52.234633 2737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:52.236116 2737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:52.245294 2756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:52.245357 2748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:52.245298 2753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:52.245365 2737 server_base.cc:1034] running on GCE node
I20260810 18:58:52.246069 2737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:52.247239 2737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:52.248454 2737 hybrid_clock.cc:648] HybridClock initialized: now 1786388332248417 us; error 45 us; skew 500 ppm
I20260810 18:58:52.250350 2737 webserver.cc:460] Webserver started at http://127.27.147.190:45109/ using document root <none> and password file <none>
I20260810 18:58:52.250988 2737 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:52.251083 2737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:52.251343 2737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:52.253818 2737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "04d2a8bb2cb24498ac74c10ae09125fd"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "1541c581b2c5f200c4f8ab054df8265d"
server_key_iv: "eb27efb4cacf641ec3cf7c2efb18c7ca"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.254405 2737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "04d2a8bb2cb24498ac74c10ae09125fd"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "1541c581b2c5f200c4f8ab054df8265d"
server_key_iv: "eb27efb4cacf641ec3cf7c2efb18c7ca"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.258126 2737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:58:52.260607 2766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.261802 2737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:52.261958 2737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "04d2a8bb2cb24498ac74c10ae09125fd"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "1541c581b2c5f200c4f8ab054df8265d"
server_key_iv: "eb27efb4cacf641ec3cf7c2efb18c7ca"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.262072 2737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:52.267071 2737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:52.267685 2737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:52.267874 2737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:52.275703 2737 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37543
I20260810 18:58:52.275728 2850 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37543 every 8 connection(s)
I20260810 18:58:52.276623 2737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:52.279829 2851 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:52.282497 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 2737
I20260810 18:58:52.282632 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:52.282985 28238 external_mini_cluster.cc:1428] Setting key 3f6befab98efd82aeed2812f67d20c77
I20260810 18:58:52.285861 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd: Bootstrap starting.
I20260810 18:58:52.288297 2851 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:52.288707 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.284436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36118 (local address 127.27.147.190:37543)
0810 18:58:52.285122 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.285138 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.285173 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:58:52.285812 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.285864 (+ 52us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.286260 (+ 396us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.286590 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.287380 (+ 790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.287385 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.287400 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.287411 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.287479 (+ 68us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.287496 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.287641 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.287715 (+ 74us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.287972 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":147,"threads_started":1}
I20260810 18:58:52.289104 2851 log.cc:824] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:52.291126 2851 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd: No bootstrap required, opened a new log
I20260810 18:58:52.295168 2851 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37543 } }
I20260810 18:58:52.295382 2851 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:52.295455 2851 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04d2a8bb2cb24498ac74c10ae09125fd, State: Initialized, Role: FOLLOWER
I20260810 18:58:52.295913 2851 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37543 } }
I20260810 18:58:52.296254 2851 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:52.296343 2851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:52.296459 2851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:52.330279 2851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37543 } }
I20260810 18:58:52.330870 2851 leader_election.cc:302] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04d2a8bb2cb24498ac74c10ae09125fd; no voters:
I20260810 18:58:52.331318 2851 leader_election.cc:288] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:52.331472 2861 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:52.331727 2861 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [term 1 LEADER]: Becoming Leader. State: Replica: 04d2a8bb2cb24498ac74c10ae09125fd, State: Running, Role: LEADER
I20260810 18:58:52.332180 2861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37543 } }
I20260810 18:58:52.332821 2851 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:52.334107 2865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04d2a8bb2cb24498ac74c10ae09125fd. Latest consensus state: current_term: 1 leader_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37543 } } }
I20260810 18:58:52.334256 2865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:52.334607 2864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04d2a8bb2cb24498ac74c10ae09125fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37543 } } }
I20260810 18:58:52.334712 2864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:52.335361 2873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:52.338339 2873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:52.344208 2873 catalog_manager.cc:1347] Generated new cluster ID: 16f2925a732343d3a3c57987aad420a1
I20260810 18:58:52.344280 2873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:52.372334 2873 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:52.372709 2873 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:52.384788 2873 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd: Generated new TSK 0
I20260810 18:58:52.385500 2873 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:58:52.393213 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37543
--builtin_ntp_servers=127.27.147.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:58:52.515326 2889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:52.515602 2889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:52.515645 2889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:52.519621 2889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:52.519698 2889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:52.519793 2889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:52.524488 2889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:52.525614 2889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:52.527092 2889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:52.537101 2903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:52.537097 2898 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:52.537096 2899 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:52.537484 2889 server_base.cc:1034] running on GCE node
I20260810 18:58:52.537951 2889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:52.539112 2889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:52.540272 2889 hybrid_clock.cc:648] HybridClock initialized: now 1786388332540237 us; error 46 us; skew 500 ppm
I20260810 18:58:52.542059 2889 webserver.cc:460] Webserver started at http://127.27.147.129:40459/ using document root <none> and password file <none>
I20260810 18:58:52.542642 2889 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:52.542706 2889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:52.542999 2889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:52.545087 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "5af8638bd924473292ecfe1c81904ea4"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "fa5ff1e79232d161a814eed917f5a0d8"
server_key_iv: "83ca78bda1d2700fca0b4aedfcb858ef"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.545670 2889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "5af8638bd924473292ecfe1c81904ea4"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "fa5ff1e79232d161a814eed917f5a0d8"
server_key_iv: "83ca78bda1d2700fca0b4aedfcb858ef"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.549515 2889 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 18:58:52.552150 2916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.553467 2889 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:52.553607 2889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "5af8638bd924473292ecfe1c81904ea4"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "fa5ff1e79232d161a814eed917f5a0d8"
server_key_iv: "83ca78bda1d2700fca0b4aedfcb858ef"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.553725 2889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:52.558728 2889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:52.559355 2889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:52.559546 2889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:52.560132 2889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:52.561061 2889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:52.561134 2889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.561204 2889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:52.561249 2889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.570338 2889 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:34111
I20260810 18:58:52.570399 3082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:34111 every 8 connection(s)
I20260810 18:58:52.571797 2889 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:52.578025 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.573699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:35349 (local address 127.27.147.190:37543)
0810 18:58:52.573845 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.573849 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.575008 (+ 1159us) server_negotiation.cc:407] Connection header received
0810 18:58:52.576438 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.576441 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.576489 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.576566 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.577635 (+ 1069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.577638 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.577640 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.577653 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.577673 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.577681 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.577755 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.577879 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.577917 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:58:52.578975 3087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.574099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37543 (local address 127.27.147.129:35349)
0810 18:58:52.574761 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.574803 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:52.576231 (+ 1428us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:52.576721 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:52.576727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:52.576813 (+ 86us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:52.577333 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:52.577335 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:52.577376 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:52.577456 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:52.577757 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:52.577764 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:52.577768 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:58:52.577899 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 18:58:52.578119 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":118,"threads_started":1}
I20260810 18:58:52.580247 3083 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37543
I20260810 18:58:52.580503 3083 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:52.580945 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 2889
I20260810 18:58:52.581058 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:52.581023 3083 heartbeater.cc:511] Master 127.27.147.190:37543 requested a full tablet report, sending...
I20260810 18:58:52.581336 28238 external_mini_cluster.cc:1428] Setting key d075dbcdb818fb4b823ec4f33ddf8af2
I20260810 18:58:52.582540 2794 ts_manager.cc:194] Registered new tserver with Master: 5af8638bd924473292ecfe1c81904ea4 (127.27.147.129:34111)
I20260810 18:58:52.583362 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37543
--builtin_ntp_servers=127.27.147.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:58:52.696036 3088 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:52.696276 3088 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:52.696313 3088 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:52.700301 3088 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:52.700348 3088 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:52.700402 3088 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:52.704909 3088 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3088
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:52.705861 3088 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:52.707376 3088 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:52.717052 3097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:52.717087 3093 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:52.717146 3088 server_base.cc:1034] running on GCE node
W20260810 18:58:52.717052 3094 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:52.717739 3088 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:52.718330 3088 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:52.719504 3088 hybrid_clock.cc:648] HybridClock initialized: now 1786388332719474 us; error 40 us; skew 500 ppm
I20260810 18:58:52.721510 3088 webserver.cc:460] Webserver started at http://127.27.147.130:36523/ using document root <none> and password file <none>
I20260810 18:58:52.722232 3088 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:52.722363 3088 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:52.722631 3088 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:52.725435 3088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "6bce2422b21148c0af79691b5533b15c"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "55405fe11a73b82ff5cf458833292ff2"
server_key_iv: "5577276d8af33c48e05df82b271b8499"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.726125 3088 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "6bce2422b21148c0af79691b5533b15c"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "55405fe11a73b82ff5cf458833292ff2"
server_key_iv: "5577276d8af33c48e05df82b271b8499"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.730369 3088 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:58:52.732964 3104 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.734344 3088 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:58:52.734445 3088 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "6bce2422b21148c0af79691b5533b15c"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "55405fe11a73b82ff5cf458833292ff2"
server_key_iv: "5577276d8af33c48e05df82b271b8499"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.734572 3088 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:52.739097 3088 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:52.739764 3088 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:52.739897 3088 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:52.740365 3088 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:52.741286 3088 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:52.741335 3088 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.741372 3088 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:52.741394 3088 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.750154 3088 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:46699
I20260810 18:58:52.750223 3282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:46699 every 8 connection(s)
I20260810 18:58:52.751096 3088 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:52.756994 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.752907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:33763 (local address 127.27.147.190:37543)
0810 18:58:52.753076 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.753081 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.754101 (+ 1020us) server_negotiation.cc:407] Connection header received
0810 18:58:52.755426 (+ 1325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.755429 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.755468 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.755547 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.756604 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.756606 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.756608 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.756617 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.756633 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.756640 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.756708 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.756818 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.756855 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 18:58:52.758215 3287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.753307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37543 (local address 127.27.147.130:33763)
0810 18:58:52.753946 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.753986 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:52.755235 (+ 1249us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:52.755702 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:52.755708 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:52.755791 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:52.756368 (+ 577us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:52.756370 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:52.756407 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:52.756482 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:52.756709 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:52.756713 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:52.756717 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:58:52.756840 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 18:58:52.757144 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":126,"threads_started":1}
I20260810 18:58:52.759402 3283 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37543
I20260810 18:58:52.759637 3283 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:52.760084 3283 heartbeater.cc:511] Master 127.27.147.190:37543 requested a full tablet report, sending...
I20260810 18:58:52.760169 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 3088
I20260810 18:58:52.760281 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:52.760571 28238 external_mini_cluster.cc:1428] Setting key 7f6a75cb30599205dfe56fa2190305d8
I20260810 18:58:52.761224 2794 ts_manager.cc:194] Registered new tserver with Master: 6bce2422b21148c0af79691b5533b15c (127.27.147.130:46699)
I20260810 18:58:52.762727 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37543
--builtin_ntp_servers=127.27.147.148:37957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 18:58:52.877709 3288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:52.878028 3288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:52.878084 3288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:52.883008 3288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:52.883067 3288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:52.883123 3288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:52.887850 3288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:37957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37543
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:52.888860 3288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:52.890234 3288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:52.900118 3293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:52.900098 3295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:52.900098 3297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:52.900252 3288 server_base.cc:1034] running on GCE node
I20260810 18:58:52.900841 3288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:52.901448 3288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:52.902643 3288 hybrid_clock.cc:648] HybridClock initialized: now 1786388332902629 us; error 26 us; skew 500 ppm
I20260810 18:58:52.904523 3288 webserver.cc:460] Webserver started at http://127.27.147.131:35115/ using document root <none> and password file <none>
I20260810 18:58:52.905102 3288 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:52.905196 3288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:52.905426 3288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:52.907714 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "21d7c5c6adca4d8698afbc62138beaf7"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "3dc31a47bcf70abcdbca3e2046617f43"
server_key_iv: "7226f30585be8914afcef1efc07e6794"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.908277 3288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "21d7c5c6adca4d8698afbc62138beaf7"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "3dc31a47bcf70abcdbca3e2046617f43"
server_key_iv: "7226f30585be8914afcef1efc07e6794"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.911888 3288 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:58:52.914304 3305 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.915503 3288 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:52.915637 3288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "21d7c5c6adca4d8698afbc62138beaf7"
format_stamp: "Formatted at 2026-08-10 18:58:52 on dist-test-slave-znh6"
server_key: "3dc31a47bcf70abcdbca3e2046617f43"
server_key_iv: "7226f30585be8914afcef1efc07e6794"
server_key_version: "encryptionkey@0"
I20260810 18:58:52.915746 3288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:52.920132 3288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:52.920697 3288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:52.920878 3288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:52.921387 3288 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:52.922266 3288 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:52.922345 3288 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.922434 3288 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:52.922488 3288 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:52.932576 3288 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:46353
I20260810 18:58:52.932623 3479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:46353 every 8 connection(s)
I20260810 18:58:52.933601 3288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:52.940039 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.935696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:49281 (local address 127.27.147.190:37543)
0810 18:58:52.935868 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.935874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.936893 (+ 1019us) server_negotiation.cc:407] Connection header received
0810 18:58:52.938153 (+ 1260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.938156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.938196 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.938270 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.939641 (+ 1371us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.939644 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.939646 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.939658 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.939678 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.939687 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.939765 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.939889 (+ 124us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.939932 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:58:52.940898 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.936088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37543 (local address 127.27.147.131:49281)
0810 18:58:52.936748 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.936786 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:52.937965 (+ 1179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:52.938501 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:52.938508 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:52.938593 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:52.939275 (+ 682us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:52.939276 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:52.939316 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:52.939392 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:52.939764 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:52.939771 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:52.939776 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:58:52.939905 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 18:58:52.940118 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":110,"threads_started":1}
I20260810 18:58:52.941133 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 3288
I20260810 18:58:52.941254 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:52.941608 28238 external_mini_cluster.cc:1428] Setting key 17e9306d96dd2096f1e0140a6c4b5569
I20260810 18:58:52.942126 3481 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37543
I20260810 18:58:52.942380 3481 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:52.942942 3481 heartbeater.cc:511] Master 127.27.147.190:37543 requested a full tablet report, sending...
I20260810 18:58:52.944013 2793 ts_manager.cc:194] Registered new tserver with Master: 21d7c5c6adca4d8698afbc62138beaf7 (127.27.147.131:46353)
I20260810 18:58:52.956014 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:58:52.959942 2855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.958174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36132 (local address 127.27.147.190:37543)
0810 18:58:52.958358 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.958363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.958513 (+ 150us) server_negotiation.cc:407] Connection header received
0810 18:58:52.958681 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.958684 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.958730 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.958816 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.959507 (+ 691us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.959510 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.959513 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.959525 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.959547 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.959558 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.959655 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.959761 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.959803 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 18:58:52.962875 2793 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36132:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:58:52.966035 2793 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:58:52.978536 3500 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.975127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:46353 (local address 127.0.0.1:48154)
0810 18:58:52.975932 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.975982 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:52.976097 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:52.977034 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:52.977042 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:52.977123 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:52.977704 (+ 581us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:52.977707 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:52.977731 (+ 24us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:52.977806 (+ 75us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:52.978179 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:52.978191 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:52.978197 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 18:58:52.978304 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 18:58:52.978375 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":91,"threads_started":1}
I20260810 18:58:52.978706 3505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.976185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:46699 (local address 127.0.0.1:41820)
0810 18:58:52.976420 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.976433 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:52.976496 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:52.977396 (+ 900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:52.977399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:52.977415 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:52.978039 (+ 624us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:52.978041 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:52.978058 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:52.978080 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:52.978458 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:52.978461 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:52.978464 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:58:52.978522 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 18:58:52.978560 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":98,"threads_started":1}
I20260810 18:58:52.979235 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.976608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41820 (local address 127.27.147.130:46699)
0810 18:58:52.976970 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.976986 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.977009 (+ 23us) server_negotiation.cc:407] Connection header received
0810 18:58:52.977065 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.977072 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.977230 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.977639 (+ 409us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.978256 (+ 617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.978261 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.978272 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.978283 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.978321 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.978350 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.978947 (+ 597us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.979004 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.979086 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":124,"threads_started":1}
I20260810 18:58:52.979245 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.975191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:34111 (local address 127.0.0.1:41940)
0810 18:58:52.976129 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:52.976142 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:52.976298 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:52.977409 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:52.977411 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:52.977421 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:52.978257 (+ 836us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:52.978259 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:52.978270 (+ 11us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:52.978310 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:52.979026 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:52.979029 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:52.979032 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:58:52.979105 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 18:58:52.979136 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":98,"threads_started":1}
I20260810 18:58:52.979246 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.975268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48154 (local address 127.27.147.131:46353)
0810 18:58:52.975979 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.975992 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.976208 (+ 216us) server_negotiation.cc:407] Connection header received
0810 18:58:52.976278 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.976286 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.976490 (+ 204us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.976631 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.977952 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.977961 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.977976 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.977990 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.978038 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.978071 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.978192 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.979020 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.979095 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":159,"threads_started":1}
I20260810 18:58:52.979372 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:52.975431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41940 (local address 127.27.147.129:34111)
0810 18:58:52.975788 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:52.975802 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:52.976222 (+ 420us) server_negotiation.cc:407] Connection header received
0810 18:58:52.976437 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:52.976446 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:52.976632 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:52.976715 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:52.978461 (+ 1746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:52.978466 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:52.978476 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:52.978490 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:52.978534 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:52.978567 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:52.978640 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:52.979125 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 18:58:52.979184 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":122,"threads_started":1}
I20260810 18:58:52.982360 3366 tablet_service.cc:1467] Processing CreateTablet for tablet 367744ad261e4198b98890e206872a4f (DEFAULT_TABLE table=test-table [id=1cb6b23e3c4d4acc862702f78ea16cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:52.982355 3181 tablet_service.cc:1467] Processing CreateTablet for tablet 367744ad261e4198b98890e206872a4f (DEFAULT_TABLE table=test-table [id=1cb6b23e3c4d4acc862702f78ea16cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:52.982648 2984 tablet_service.cc:1467] Processing CreateTablet for tablet 367744ad261e4198b98890e206872a4f (DEFAULT_TABLE table=test-table [id=1cb6b23e3c4d4acc862702f78ea16cdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:58:52.983572 3366 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367744ad261e4198b98890e206872a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:52.983572 3181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367744ad261e4198b98890e206872a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:52.983690 2984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 367744ad261e4198b98890e206872a4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:52.989645 3507 tablet_bootstrap.cc:492] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4: Bootstrap starting.
I20260810 18:58:52.991341 3509 tablet_bootstrap.cc:492] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Bootstrap starting.
I20260810 18:58:52.992410 3507 tablet_bootstrap.cc:654] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:52.993309 3507 log.cc:824] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:52.993489 3509 tablet_bootstrap.cc:654] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:52.994009 3508 tablet_bootstrap.cc:492] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Bootstrap starting.
I20260810 18:58:52.994345 3509 log.cc:824] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:52.995462 3507 tablet_bootstrap.cc:492] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4: No bootstrap required, opened a new log
I20260810 18:58:52.995672 3507 ts_tablet_manager.cc:1397] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:58:52.996116 3508 tablet_bootstrap.cc:654] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:52.996376 3509 tablet_bootstrap.cc:492] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: No bootstrap required, opened a new log
I20260810 18:58:52.996564 3509 ts_tablet_manager.cc:1397] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:58:52.996942 3508 log.cc:824] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:52.999007 3508 tablet_bootstrap.cc:492] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: No bootstrap required, opened a new log
I20260810 18:58:52.999214 3508 ts_tablet_manager.cc:1397] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.002s
I20260810 18:58:53.000221 3509 raft_consensus.cc:348] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.000465 3509 raft_consensus.cc:374] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:53.000520 3509 raft_consensus.cc:729] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21d7c5c6adca4d8698afbc62138beaf7, State: Initialized, Role: FOLLOWER
I20260810 18:58:53.000428 3507 raft_consensus.cc:348] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.000679 3507 raft_consensus.cc:374] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:53.000733 3507 raft_consensus.cc:729] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5af8638bd924473292ecfe1c81904ea4, State: Initialized, Role: FOLLOWER
I20260810 18:58:53.001055 3509 consensus_queue.cc:260] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.001152 3507 consensus_queue.cc:260] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.001677 3481 heartbeater.cc:503] Master 127.27.147.190:37543 was elected leader, sending a full tablet report...
I20260810 18:58:53.002172 3509 ts_tablet_manager.cc:1428] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Time spent starting tablet: real 0.006s user 0.000s sys 0.007s
I20260810 18:58:53.002208 3507 ts_tablet_manager.cc:1428] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 18:58:53.002588 3083 heartbeater.cc:503] Master 127.27.147.190:37543 was elected leader, sending a full tablet report...
I20260810 18:58:53.003796 3508 raft_consensus.cc:348] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.004035 3508 raft_consensus.cc:374] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:53.004076 3508 raft_consensus.cc:729] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6bce2422b21148c0af79691b5533b15c, State: Initialized, Role: FOLLOWER
I20260810 18:58:53.004485 3508 consensus_queue.cc:260] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.005182 3283 heartbeater.cc:503] Master 127.27.147.190:37543 was elected leader, sending a full tablet report...
I20260810 18:58:53.005501 3508 ts_tablet_manager.cc:1428] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
W20260810 18:58:53.073048 3086 tablet.cc:2369] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:58:53.184688 3482 tablet.cc:2369] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:58:53.210875 3514 raft_consensus.cc:482] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:58:53.211081 3514 raft_consensus.cc:504] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.212167 3514 leader_election.cc:288] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5af8638bd924473292ecfe1c81904ea4 (127.27.147.129:34111), 6bce2422b21148c0af79691b5533b15c (127.27.147.130:46699)
I20260810 18:58:53.214161 3504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.212576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:49453 (local address 127.27.147.129:34111)
0810 18:58:53.212747 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:53.212754 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:53.212772 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:58:53.212845 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:53.212849 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:53.212892 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:53.212997 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:53.213781 (+ 784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:53.213784 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:53.213787 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:53.213805 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:53.213821 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:53.213829 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:53.213926 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:53.214001 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 18:58:53.214037 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 18:58:53.214160 3483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.212433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:34111 (local address 127.27.147.131:49453)
0810 18:58:53.212600 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:53.212619 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:53.212710 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:53.213031 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:53.213034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:53.213048 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:53.213622 (+ 574us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:53.213623 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:53.213633 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:53.213657 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:53.213910 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:53.213913 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:53.213916 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:58:53.213994 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:58:53.214033 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 18:58:53.214359 3506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.212580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:46377 (local address 127.27.147.130:46699)
0810 18:58:53.212746 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:53.212753 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:53.213133 (+ 380us) server_negotiation.cc:407] Connection header received
0810 18:58:53.213212 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:53.213217 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:53.213261 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:53.213334 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:53.213988 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:53.213992 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:53.213994 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:53.214006 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:53.214032 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:53.214040 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:53.214136 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:53.214213 (+ 77us) server_negotiation.cc:299] Negotiation successful
0810 18:58:53.214248 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:58:53.214380 3520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.212433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:46699 (local address 127.27.147.131:46377)
0810 18:58:53.213024 (+ 591us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:53.213040 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:53.213131 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:53.213379 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:53.213383 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:53.213399 (+ 16us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:53.213808 (+ 409us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:58:53.213809 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:53.213819 (+ 10us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:58:53.213843 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:53.214120 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:53.214123 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:53.214126 (+ 3us) client_negotiation.cc:769] Sending connection context
0810 18:58:53.214204 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:58:53.214241 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":228,"thread_start_us":125,"threads_started":1}
I20260810 18:58:53.214890 3016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367744ad261e4198b98890e206872a4f" candidate_uuid: "21d7c5c6adca4d8698afbc62138beaf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af8638bd924473292ecfe1c81904ea4" is_pre_election: true
I20260810 18:58:53.215103 3214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367744ad261e4198b98890e206872a4f" candidate_uuid: "21d7c5c6adca4d8698afbc62138beaf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bce2422b21148c0af79691b5533b15c" is_pre_election: true
I20260810 18:58:53.215211 3016 raft_consensus.cc:2454] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d7c5c6adca4d8698afbc62138beaf7 in term 0.
I20260810 18:58:53.215395 3214 raft_consensus.cc:2454] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21d7c5c6adca4d8698afbc62138beaf7 in term 0.
I20260810 18:58:53.215725 3310 leader_election.cc:302] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d7c5c6adca4d8698afbc62138beaf7, 5af8638bd924473292ecfe1c81904ea4; no voters:
I20260810 18:58:53.216017 3514 raft_consensus.cc:2781] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:58:53.216091 3514 raft_consensus.cc:482] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:58:53.216122 3514 raft_consensus.cc:3037] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:53.217252 3514 raft_consensus.cc:504] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.217597 3514 leader_election.cc:288] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [CANDIDATE]: Term 1 election: Requested vote from peers 5af8638bd924473292ecfe1c81904ea4 (127.27.147.129:34111), 6bce2422b21148c0af79691b5533b15c (127.27.147.130:46699)
I20260810 18:58:53.217933 3214 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367744ad261e4198b98890e206872a4f" candidate_uuid: "21d7c5c6adca4d8698afbc62138beaf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6bce2422b21148c0af79691b5533b15c"
I20260810 18:58:53.217975 3016 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "367744ad261e4198b98890e206872a4f" candidate_uuid: "21d7c5c6adca4d8698afbc62138beaf7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5af8638bd924473292ecfe1c81904ea4"
I20260810 18:58:53.218043 3214 raft_consensus.cc:3037] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:53.218124 3016 raft_consensus.cc:3037] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:53.219246 3214 raft_consensus.cc:2454] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d7c5c6adca4d8698afbc62138beaf7 in term 1.
I20260810 18:58:53.219419 3016 raft_consensus.cc:2454] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21d7c5c6adca4d8698afbc62138beaf7 in term 1.
I20260810 18:58:53.219597 3306 leader_election.cc:302] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21d7c5c6adca4d8698afbc62138beaf7, 6bce2422b21148c0af79691b5533b15c; no voters:
I20260810 18:58:53.219774 3514 raft_consensus.cc:2781] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:53.220021 3514 raft_consensus.cc:686] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 1 LEADER]: Becoming Leader. State: Replica: 21d7c5c6adca4d8698afbc62138beaf7, State: Running, Role: LEADER
I20260810 18:58:53.220361 3514 consensus_queue.cc:237] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } }
I20260810 18:58:53.223502 2791 catalog_manager.cc:5495] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 21d7c5c6adca4d8698afbc62138beaf7 (127.27.147.131). New cstate: current_term: 1 leader_uuid: "21d7c5c6adca4d8698afbc62138beaf7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21d7c5c6adca4d8698afbc62138beaf7" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 46353 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 } health_report { overall_health: UNKNOWN } } }
W20260810 18:58:53.252221 3286 tablet.cc:2369] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:58:53.278049 3502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.270306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48156 (local address 127.27.147.131:46353)
0810 18:58:53.270439 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:53.270445 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:53.270460 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:58:53.270568 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:53.270570 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:53.270632 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:53.270730 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:53.277569 (+ 6839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:53.277573 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:53.277575 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:53.277588 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:53.277610 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:53.277619 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:53.277710 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:53.277817 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 18:58:53.277858 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:58:53.284343 3214 raft_consensus.cc:1261] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 1 FOLLOWER]: Refusing update from remote peer 21d7c5c6adca4d8698afbc62138beaf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:53.284353 3016 raft_consensus.cc:1261] T 367744ad261e4198b98890e206872a4f P 5af8638bd924473292ecfe1c81904ea4 [term 1 FOLLOWER]: Refusing update from remote peer 21d7c5c6adca4d8698afbc62138beaf7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:58:53.285094 3514 consensus_queue.cc:1035] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6bce2422b21148c0af79691b5533b15c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 46699 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:53.285295 3521 consensus_queue.cc:1035] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5af8638bd924473292ecfe1c81904ea4" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 34111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:58:53.292495 3524 mvcc.cc:205] Tried to move back new op lower bound from 7317046613125001216 to 7317046612873068544. Current Snapshot: MvccSnapshot[applied={T|T < 7317046613125001216}]
I20260810 18:58:53.293789 3528 mvcc.cc:205] Tried to move back new op lower bound from 7317046613125001216 to 7317046612873068544. Current Snapshot: MvccSnapshot[applied={T|T < 7317046613125001216}]
I20260810 18:58:53.294441 3527 mvcc.cc:205] Tried to move back new op lower bound from 7317046613125001216 to 7317046612873068544. Current Snapshot: MvccSnapshot[applied={T|T < 7317046613125001216}]
I20260810 18:58:53.301223 2793 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:53.301364 2793 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36132:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:58:53.303864 2793 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 04d2a8bb2cb24498ac74c10ae09125fd: Sending DeleteTablet for 3 replicas of tablet 367744ad261e4198b98890e206872a4f
I20260810 18:58:53.304636 3366 tablet_service.cc:1514] Processing DeleteTablet for tablet 367744ad261e4198b98890e206872a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:53 UTC) from {username='slave'} at 127.0.0.1:48154
I20260810 18:58:53.304636 2984 tablet_service.cc:1514] Processing DeleteTablet for tablet 367744ad261e4198b98890e206872a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:53 UTC) from {username='slave'} at 127.0.0.1:41940
I20260810 18:58:53.304829 3181 tablet_service.cc:1514] Processing DeleteTablet for tablet 367744ad261e4198b98890e206872a4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:58:53 UTC) from {username='slave'} at 127.0.0.1:41820
I20260810 18:58:53.305331 3541 tablet_replica.cc:330] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: stopping tablet replica
I20260810 18:58:53.305337 3539 tablet_replica.cc:330] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: stopping tablet replica
I20260810 18:58:53.305644 3539 raft_consensus.cc:2229] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:58:53.305966 3541 raft_consensus.cc:2229] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:58:53.306026 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 2889
I20260810 18:58:53.306075 3539 raft_consensus.cc:2258] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:53.309023 3541 raft_consensus.cc:2258] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:58:53.310708 3541 ts_tablet_manager.cc:1905] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:53.313892 3541 ts_tablet_manager.cc:1918] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:58:53.313978 3541 log.cc:1196] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/367744ad261e4198b98890e206872a4f
I20260810 18:58:53.314291 3541 ts_tablet_manager.cc:1939] T 367744ad261e4198b98890e206872a4f P 6bce2422b21148c0af79691b5533b15c: Deleting consensus metadata
I20260810 18:58:53.315656 2767 catalog_manager.cc:4841] TS 6bce2422b21148c0af79691b5533b15c (127.27.147.130:46699): tablet 367744ad261e4198b98890e206872a4f (table test-table [id=1cb6b23e3c4d4acc862702f78ea16cdb]) successfully deleted
I20260810 18:58:53.316581 3539 ts_tablet_manager.cc:1905] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:58:53.319281 3539 ts_tablet_manager.cc:1918] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:58:53.319380 3539 log.cc:1196] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/367744ad261e4198b98890e206872a4f
I20260810 18:58:53.319769 3539 ts_tablet_manager.cc:1939] T 367744ad261e4198b98890e206872a4f P 21d7c5c6adca4d8698afbc62138beaf7: Deleting consensus metadata
I20260810 18:58:53.320911 2771 catalog_manager.cc:4841] TS 21d7c5c6adca4d8698afbc62138beaf7 (127.27.147.131:46353): tablet 367744ad261e4198b98890e206872a4f (table test-table [id=1cb6b23e3c4d4acc862702f78ea16cdb]) successfully deleted
W20260810 18:58:53.321519 2772 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:34111 (error 108)
I20260810 18:58:53.322441 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 3088
I20260810 18:58:53.322650 3499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.322301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:34111 (local address 127.0.0.1:41952)
0810 18:58:53.322495 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:53.322565 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:34111: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260810 18:58:53.333719 2772 catalog_manager.cc:4584] TS 5af8638bd924473292ecfe1c81904ea4 (127.27.147.129:34111): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 367744ad261e4198b98890e206872a4f: Network error: Client connection negotiation failed: client connection to 127.27.147.129:34111: connect: Connection refused (error 111)
I20260810 18:58:53.339561 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 3288
I20260810 18:58:53.355868 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 2737
2026-08-10T18:58:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (1289 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 18:58:53.387879 28238 mini_oidc.cc:152] Starting JWKS server
I20260810 18:58:53.389683 28238 webserver.cc:460] Webserver started at https://127.0.0.1:39211/ using document root <none> and password file <none>
I20260810 18:58:53.389868 28238 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:58:53.390172 28238 webserver.cc:460] Webserver started at http://127.0.0.1:37535/ using document root <none> and password file <none>
2026-08-10T18:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:53Z Disabled control of system clock
I20260810 18:58:53.400818 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37827
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:34115
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37827
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39211/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/testchainca.cert with env {}
W20260810 18:58:53.515950 3552 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:53.516206 3552 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:53.516247 3552 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:53.520174 3552 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:53.520228 3552 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:53.520244 3552 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:53.520262 3552 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:53.520277 3552 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:58:53.520292 3552 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:58:53.520334 3552 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39211/jwks.json
I20260810 18:58:53.525702 3552 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34115
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37827
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39211/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3552
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:53.526746 3552 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:53.527803 3552 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:53.536326 3560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:53.536335 3558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:53.536404 3552 server_base.cc:1034] running on GCE node
W20260810 18:58:53.536335 3557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:53.537096 3552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:53.538240 3552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:53.539427 3552 hybrid_clock.cc:648] HybridClock initialized: now 1786388333539408 us; error 31 us; skew 500 ppm
I20260810 18:58:53.541318 3552 webserver.cc:460] Webserver started at http://127.27.147.190:35973/ using document root <none> and password file <none>
I20260810 18:58:53.541908 3552 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:53.542006 3552 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:53.542239 3552 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:53.544318 3552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "2108b67ec23f4dddbf0b260eb37c67f9"
format_stamp: "Formatted at 2026-08-10 18:58:53 on dist-test-slave-znh6"
server_key: "0275687fa7a1cd6f17e4f77af2539619"
server_key_iv: "11349e0f9c9d3bb58afc1d182d1690f7"
server_key_version: "encryptionkey@0"
I20260810 18:58:53.544883 3552 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "2108b67ec23f4dddbf0b260eb37c67f9"
format_stamp: "Formatted at 2026-08-10 18:58:53 on dist-test-slave-znh6"
server_key: "0275687fa7a1cd6f17e4f77af2539619"
server_key_iv: "11349e0f9c9d3bb58afc1d182d1690f7"
server_key_version: "encryptionkey@0"
I20260810 18:58:53.548560 3552 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:58:53.551023 3571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:53.552171 3552 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:53.552314 3552 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "2108b67ec23f4dddbf0b260eb37c67f9"
format_stamp: "Formatted at 2026-08-10 18:58:53 on dist-test-slave-znh6"
server_key: "0275687fa7a1cd6f17e4f77af2539619"
server_key_iv: "11349e0f9c9d3bb58afc1d182d1690f7"
server_key_version: "encryptionkey@0"
I20260810 18:58:53.552425 3552 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:53.571825 3552 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:53.572557 3552 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:53.572765 3552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:53.580116 3552 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37827
I20260810 18:58:53.580142 3643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37827 every 8 connection(s)
I20260810 18:58:53.581130 3552 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:53.584353 3644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:53.587075 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 3552
I20260810 18:58:53.587203 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:53.587498 28238 external_mini_cluster.cc:1428] Setting key 285f42558d8be7453dcedd50d879bc33
I20260810 18:58:53.590372 3644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9: Bootstrap starting.
I20260810 18:58:53.593000 3644 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:53.593863 3644 log.cc:824] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:53.596143 3644 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9: No bootstrap required, opened a new log
I20260810 18:58:53.596912 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.588927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48234 (local address 127.27.147.190:37827)
0810 18:58:53.589557 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:53.589574 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:53.589611 (+ 37us) server_negotiation.cc:407] Connection header received
0810 18:58:53.590280 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:53.590299 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:53.590748 (+ 449us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:53.591086 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:53.591449 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.592384 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.593633 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.594147 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.594300 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:53.595152 (+ 852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:53.595157 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:53.595169 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:53.595558 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:53.595613 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:53.595634 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:53.595712 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:53.595881 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 18:58:53.596166 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":131,"threads_started":1}
I20260810 18:58:53.599777 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:58:53.599915 3644 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37827 } }
I20260810 18:58:53.600142 3644 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:53.600216 3644 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2108b67ec23f4dddbf0b260eb37c67f9, State: Initialized, Role: FOLLOWER
I20260810 18:58:53.600629 3644 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37827 } }
I20260810 18:58:53.600809 3644 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:53.600888 3644 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:53.601018 3644 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:53.602046 3644 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37827 } }
I20260810 18:58:53.602388 3644 leader_election.cc:302] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2108b67ec23f4dddbf0b260eb37c67f9; no voters:
I20260810 18:58:53.602701 3644 leader_election.cc:288] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:53.602823 3650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:53.603113 3650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [term 1 LEADER]: Becoming Leader. State: Replica: 2108b67ec23f4dddbf0b260eb37c67f9, State: Running, Role: LEADER
I20260810 18:58:53.603492 3650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37827 } }
I20260810 18:58:53.604067 3644 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:53.605144 3653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2108b67ec23f4dddbf0b260eb37c67f9. Latest consensus state: current_term: 1 leader_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37827 } } }
I20260810 18:58:53.605294 3653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:53.605701 3664 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:53.606835 3652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2108b67ec23f4dddbf0b260eb37c67f9" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37827 } } }
I20260810 18:58:53.606933 3652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:53.608990 3664 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:53.614809 3664 catalog_manager.cc:1347] Generated new cluster ID: 3bfa65c5aa66401e8e93db8faacc0a16
I20260810 18:58:53.614933 3664 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:53.642167 3664 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:53.643234 3664 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:53.649012 3664 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2108b67ec23f4dddbf0b260eb37c67f9: Generated new TSK 0
I20260810 18:58:53.649714 3664 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:58:53.681254 3688 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 18:58:53.681420 3688 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:58:53.682014 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.664434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48238 (local address 127.27.147.190:37827)
0810 18:58:53.664576 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:53.664581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:53.664758 (+ 177us) server_negotiation.cc:407] Connection header received
0810 18:58:53.664956 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:53.664959 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:53.665016 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:53.665125 (+ 109us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:58:53.665371 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.666219 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.667161 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.667519 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.667627 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:53.667933 (+ 306us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:58:53.681598 (+ 13665us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:58:53.681862 (+ 264us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 18:58:53.682098 3648 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48238: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 18:58:53.683380 3686 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:53.664317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37827 (local address 127.0.0.1:48238)
0810 18:58:53.664676 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:53.664689 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:53.664798 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:53.665141 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:53.665143 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:53.665150 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:58:53.665267 (+ 117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:53.665272 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.666339 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.666343 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:53.667041 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:53.667050 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.667631 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.667634 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:53.667702 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:53.667746 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:58:53.681803 (+ 14057us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 18:58:53.681996 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.147.190:37827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":120,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.27.147.190:37827: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39211/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 18:58:53.684083 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 3552
2026-08-10T18:58:53Z chronyd exiting
I20260810 18:58:53.728572 28238 test_util.cc:175] -----------------------------------------------
I20260810 18:58:53.728664 28238 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786388305279399-28238-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (342 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 18:58:53.729857 28238 mini_oidc.cc:152] Starting JWKS server
I20260810 18:58:53.730746 28238 webserver.cc:460] Webserver started at https://127.0.0.1:39653/ using document root <none> and password file <none>
I20260810 18:58:53.730898 28238 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:58:53.731199 28238 webserver.cc:460] Webserver started at http://127.0.0.1:45759/ using document root <none> and password file <none>
2026-08-10T18:58:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:53Z Disabled control of system clock
I20260810 18:58:53.741693 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:33803
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:46689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:33803
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39653/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/testchainca.cert with env {}
W20260810 18:58:53.856923 3699 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:53.857206 3699 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:53.857246 3699 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:53.861214 3699 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:53.861275 3699 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:53.861292 3699 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:53.861308 3699 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:53.861325 3699 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:58:53.861340 3699 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:58:53.861354 3699 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39653/jwks.json
I20260810 18:58:53.866781 3699 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:33803
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:33803
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39653/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3699
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:53.867877 3699 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:53.868911 3699 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:53.877947 3710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:53.877951 3708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:53.877954 3713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:53.878171 3699 server_base.cc:1034] running on GCE node
I20260810 18:58:53.878966 3699 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:53.880122 3699 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:53.881287 3699 hybrid_clock.cc:648] HybridClock initialized: now 1786388333881265 us; error 35 us; skew 500 ppm
I20260810 18:58:53.883387 3699 webserver.cc:460] Webserver started at http://127.27.147.190:41783/ using document root <none> and password file <none>
I20260810 18:58:53.883997 3699 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:53.884063 3699 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:53.884296 3699 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:53.886370 3699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "da1bfb074252446ca2ea645ac891a28c"
format_stamp: "Formatted at 2026-08-10 18:58:53 on dist-test-slave-znh6"
server_key: "824a2f47f5c18cce5c85b4727741325e"
server_key_iv: "fc88b2ac2f8882aa51e4e08d809cafa8"
server_key_version: "encryptionkey@0"
I20260810 18:58:53.887017 3699 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "da1bfb074252446ca2ea645ac891a28c"
format_stamp: "Formatted at 2026-08-10 18:58:53 on dist-test-slave-znh6"
server_key: "824a2f47f5c18cce5c85b4727741325e"
server_key_iv: "fc88b2ac2f8882aa51e4e08d809cafa8"
server_key_version: "encryptionkey@0"
I20260810 18:58:53.890661 3699 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:58:53.893076 3721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:53.894301 3699 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:53.894451 3699 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "da1bfb074252446ca2ea645ac891a28c"
format_stamp: "Formatted at 2026-08-10 18:58:53 on dist-test-slave-znh6"
server_key: "824a2f47f5c18cce5c85b4727741325e"
server_key_iv: "fc88b2ac2f8882aa51e4e08d809cafa8"
server_key_version: "encryptionkey@0"
I20260810 18:58:53.894567 3699 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:53.918113 3699 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:53.918875 3699 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:53.919080 3699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:53.926343 3699 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:33803
I20260810 18:58:53.926353 3802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:33803 every 8 connection(s)
I20260810 18:58:53.927326 3699 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:53.927696 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 3699
I20260810 18:58:53.927826 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:53.928140 28238 external_mini_cluster.cc:1428] Setting key a860056ddfeba6e476af9e585d6b1874
I20260810 18:58:53.931077 3803 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:53.936609 3803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c: Bootstrap starting.
I20260810 18:58:53.937528 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:53.929573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.27.147.190:33803)
0810 18:58:53.930250 (+ 677us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:53.930269 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:53.930305 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:58:53.931062 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:53.931081 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:53.931617 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:53.931956 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:53.932420 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.933388 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.934277 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:53.934756 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:53.934971 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:53.935836 (+ 865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:53.935842 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:53.935854 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:53.936228 (+ 374us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:53.936281 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:53.936300 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:53.936409 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:53.936569 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 18:58:53.936843 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":152,"threads_started":1}
I20260810 18:58:53.938985 3803 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:53.939872 3803 log.cc:824] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:53.940389 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:58:53.942018 3803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c: No bootstrap required, opened a new log
I20260810 18:58:53.945747 3803 raft_consensus.cc:348] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bfb074252446ca2ea645ac891a28c" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33803 } }
I20260810 18:58:53.945959 3803 raft_consensus.cc:374] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:53.945995 3803 raft_consensus.cc:729] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: da1bfb074252446ca2ea645ac891a28c, State: Initialized, Role: FOLLOWER
I20260810 18:58:53.946343 3803 consensus_queue.cc:260] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bfb074252446ca2ea645ac891a28c" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33803 } }
I20260810 18:58:53.946455 3803 raft_consensus.cc:388] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:53.946504 3803 raft_consensus.cc:482] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:53.946573 3803 raft_consensus.cc:3037] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:53.947517 3803 raft_consensus.cc:504] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bfb074252446ca2ea645ac891a28c" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33803 } }
I20260810 18:58:53.947803 3803 leader_election.cc:302] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: da1bfb074252446ca2ea645ac891a28c; no voters:
I20260810 18:58:53.948048 3803 leader_election.cc:288] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:53.948172 3809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:53.948441 3809 raft_consensus.cc:686] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [term 1 LEADER]: Becoming Leader. State: Replica: da1bfb074252446ca2ea645ac891a28c, State: Running, Role: LEADER
I20260810 18:58:53.948781 3809 consensus_queue.cc:237] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bfb074252446ca2ea645ac891a28c" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33803 } }
I20260810 18:58:53.949340 3803 sys_catalog.cc:564] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:53.950590 3810 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "da1bfb074252446ca2ea645ac891a28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bfb074252446ca2ea645ac891a28c" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33803 } } }
I20260810 18:58:53.950606 3813 sys_catalog.cc:455] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [sys.catalog]: SysCatalogTable state changed. Reason: New leader da1bfb074252446ca2ea645ac891a28c. Latest consensus state: current_term: 1 leader_uuid: "da1bfb074252446ca2ea645ac891a28c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "da1bfb074252446ca2ea645ac891a28c" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33803 } } }
I20260810 18:58:53.950742 3810 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:53.950747 3813 sys_catalog.cc:458] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:53.951174 3820 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:53.953732 3820 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:53.959452 3820 catalog_manager.cc:1347] Generated new cluster ID: e9c77d310f584c8892f6bc42fc364fa1
I20260810 18:58:53.959520 3820 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:53.984839 3820 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:53.985811 3820 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:53.996040 3820 catalog_manager.cc:5868] T 00000000000000000000000000000000 P da1bfb074252446ca2ea645ac891a28c: Generated new TSK 0
I20260810 18:58:53.996794 3820 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 18:58:54.026161 3841 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 18:58:54.026268 3841 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 18:58:54.027009 3807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:54.011159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54004 (local address 127.27.147.190:33803)
0810 18:58:54.011349 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:54.011354 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:54.011481 (+ 127us) server_negotiation.cc:407] Connection header received
0810 18:58:54.011643 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:54.011646 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:54.011706 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:54.011819 (+ 113us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:58:54.012038 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.012839 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.013735 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.014082 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.014190 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.014475 (+ 285us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:58:54.026560 (+ 12085us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:58:54.026853 (+ 293us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54004: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":77}
W20260810 18:58:54.027076 3807 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54004: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 18:58:54.027158 3840 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:54.011098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:33803 (local address 127.0.0.1:54004)
0810 18:58:54.011368 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:54.011391 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:54.011498 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:54.011823 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:54.011825 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:54.011832 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:58:54.011925 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:54.011930 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.012982 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.012985 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:54.013611 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:54.013618 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.014200 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.014203 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:54.014270 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.014294 (+ 24us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:58:54.026761 (+ 12467us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 18:58:54.026968 (+ 207us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.147.190:33803: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39653/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260810 18:58:54.027570 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 3699
2026-08-10T18:58:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (349 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 18:58:54.078979 28238 mini_oidc.cc:152] Starting JWKS server
I20260810 18:58:54.079890 28238 webserver.cc:460] Webserver started at https://127.0.0.1:43009/ using document root <none> and password file <none>
I20260810 18:58:54.080034 28238 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 18:58:54.080312 28238 webserver.cc:460] Webserver started at http://127.0.0.1:41717/ using document root <none> and password file <none>
2026-08-10T18:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:54Z Disabled control of system clock
I20260810 18:58:54.091233 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37161
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:33759
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37161
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43009/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:58:54.208509 3849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:54.208760 3849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:54.208799 3849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:54.212975 3849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:54.213032 3849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:54.213047 3849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:54.213063 3849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:54.213104 3849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:58:54.213135 3849 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:58:54.213161 3849 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43009/jwks.json
I20260810 18:58:54.217907 3849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:33759
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37161
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37161
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43009/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:54.219008 3849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:54.220413 3849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:54.229111 3857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:54.229104 3859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:54.229161 3849 server_base.cc:1034] running on GCE node
W20260810 18:58:54.229104 3861 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:54.229864 3849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:54.231001 3849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:54.232160 3849 hybrid_clock.cc:648] HybridClock initialized: now 1786388334232131 us; error 42 us; skew 500 ppm
I20260810 18:58:54.234128 3849 webserver.cc:460] Webserver started at http://127.27.147.190:36481/ using document root <none> and password file <none>
I20260810 18:58:54.234725 3849 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:54.234789 3849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:54.235064 3849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:54.237130 3849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "ad4ebb8997db4679903ac2ed2f8bc546"
format_stamp: "Formatted at 2026-08-10 18:58:54 on dist-test-slave-znh6"
server_key: "279a8b6a481596af34fe66d594971440"
server_key_iv: "7180b34d21f8ed5e431e212c2c7ffb91"
server_key_version: "encryptionkey@0"
I20260810 18:58:54.237694 3849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "ad4ebb8997db4679903ac2ed2f8bc546"
format_stamp: "Formatted at 2026-08-10 18:58:54 on dist-test-slave-znh6"
server_key: "279a8b6a481596af34fe66d594971440"
server_key_iv: "7180b34d21f8ed5e431e212c2c7ffb91"
server_key_version: "encryptionkey@0"
I20260810 18:58:54.241377 3849 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 18:58:54.243803 3867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:54.244971 3849 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:58:54.245100 3849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "ad4ebb8997db4679903ac2ed2f8bc546"
format_stamp: "Formatted at 2026-08-10 18:58:54 on dist-test-slave-znh6"
server_key: "279a8b6a481596af34fe66d594971440"
server_key_iv: "7180b34d21f8ed5e431e212c2c7ffb91"
server_key_version: "encryptionkey@0"
I20260810 18:58:54.245204 3849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:54.259080 3849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:54.259763 3849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:54.259963 3849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:54.267400 3849 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37161
I20260810 18:58:54.267416 3945 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37161 every 8 connection(s)
I20260810 18:58:54.268405 3849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:54.271716 3946 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:54.277304 3946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546: Bootstrap starting.
I20260810 18:58:54.278146 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 3849
I20260810 18:58:54.278265 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:54.278594 28238 external_mini_cluster.cc:1428] Setting key 0db0a140623fbc851ed44cffbebd3e6a
I20260810 18:58:54.280440 3946 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:54.281271 3946 log.cc:824] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:54.283412 3946 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546: No bootstrap required, opened a new log
I20260810 18:58:54.287750 3946 raft_consensus.cc:348] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37161 } }
I20260810 18:58:54.287969 3946 raft_consensus.cc:374] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:54.288014 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:54.279807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34084 (local address 127.27.147.190:37161)
0810 18:58:54.280530 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:54.280553 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:54.280608 (+ 55us) server_negotiation.cc:407] Connection header received
0810 18:58:54.281268 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:54.281285 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:54.281808 (+ 523us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:54.282154 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:54.282501 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.283573 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.284741 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.285197 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.285311 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.286160 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:54.286165 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:54.286177 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:54.286565 (+ 388us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:54.286622 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:54.286642 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:54.286717 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:54.286950 (+ 233us) server_negotiation.cc:299] Negotiation successful
0810 18:58:54.287252 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":143,"threads_started":1}
I20260810 18:58:54.288029 3946 raft_consensus.cc:729] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ad4ebb8997db4679903ac2ed2f8bc546, State: Initialized, Role: FOLLOWER
I20260810 18:58:54.288558 3946 consensus_queue.cc:260] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37161 } }
I20260810 18:58:54.288686 3946 raft_consensus.cc:388] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:54.288772 3946 raft_consensus.cc:482] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:54.288911 3946 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:54.289775 3946 raft_consensus.cc:504] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37161 } }
I20260810 18:58:54.290151 3946 leader_election.cc:302] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ad4ebb8997db4679903ac2ed2f8bc546; no voters:
I20260810 18:58:54.290535 3946 leader_election.cc:288] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:54.290577 3962 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:54.290755 3962 raft_consensus.cc:686] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [term 1 LEADER]: Becoming Leader. State: Replica: ad4ebb8997db4679903ac2ed2f8bc546, State: Running, Role: LEADER
I20260810 18:58:54.290858 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:58:54.291165 3962 consensus_queue.cc:237] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37161 } }
I20260810 18:58:54.291810 3946 sys_catalog.cc:564] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:54.292954 3964 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37161 } } }
I20260810 18:58:54.293116 3964 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:54.293433 3963 sys_catalog.cc:455] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ad4ebb8997db4679903ac2ed2f8bc546. Latest consensus state: current_term: 1 leader_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ad4ebb8997db4679903ac2ed2f8bc546" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37161 } } }
I20260810 18:58:54.293524 3963 sys_catalog.cc:458] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:54.293601 3975 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:54.296574 3975 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:54.302067 3975 catalog_manager.cc:1347] Generated new cluster ID: 72d261fb5c2b4cb2b8b74e058b6f8138
I20260810 18:58:54.302146 3975 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:54.314018 3975 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:54.314960 3975 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:54.322057 3975 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ad4ebb8997db4679903ac2ed2f8bc546: Generated new TSK 0
I20260810 18:58:54.322737 3975 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:58:54.336647 3958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:54.331524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.27.147.190:37161)
0810 18:58:54.331662 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:54.331667 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:54.331852 (+ 185us) server_negotiation.cc:407] Connection header received
0810 18:58:54.332015 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:54.332017 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:54.332062 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:54.332150 (+ 88us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:58:54.332407 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.332941 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.334186 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.334403 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.334494 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.334806 (+ 312us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:58:54.336241 (+ 1435us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:58:54.336491 (+ 250us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:34098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 18:58:54.336722 3958 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:34098: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 18:58:54.336730 3996 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:54.331438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37161 (local address 127.0.0.1:34098)
0810 18:58:54.331758 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:54.331774 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:54.331894 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:54.332175 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:54.332179 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:54.332187 (+ 8us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:58:54.332303 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:54.332310 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.333051 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.333055 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:54.334053 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:54.334063 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.334531 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.334535 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:54.334625 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.334654 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:58:54.336406 (+ 1752us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 18:58:54.336556 (+ 150us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.147.190:37161: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43009/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":123,"threads_started":1}
I20260810 18:58:54.337410 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 3849
E20260810 18:58:54.359766 3997 webserver.cc:533] Webserver: sslize failed:
E20260810 18:58:54.359853 3997 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-08-10T18:58:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (313 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T18:58:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:54Z Disabled control of system clock
I20260810 18:58:54.404361 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:36763
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:44049
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:36763
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/testchainca.cert with env {}
W20260810 18:58:54.520133 4003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:54.520407 4003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:54.520448 4003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:54.524462 4003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:54.524519 4003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:54.524534 4003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:54.524573 4003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:54.524608 4003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 18:58:54.524639 4003 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 18:58:54.524654 4003 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 18:58:54.530049 4003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:44049
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:36763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:36763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4003
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:54.531149 4003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:54.532137 4003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:54.540658 4016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:54.540867 4003 server_base.cc:1034] running on GCE node
W20260810 18:58:54.540678 4020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:54.540671 4018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:54.541513 4003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:54.542598 4003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:54.543795 4003 hybrid_clock.cc:648] HybridClock initialized: now 1786388334543771 us; error 41 us; skew 500 ppm
I20260810 18:58:54.545737 4003 webserver.cc:460] Webserver started at http://127.27.147.190:42575/ using document root <none> and password file <none>
I20260810 18:58:54.546331 4003 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:54.546448 4003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:54.546690 4003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:54.548801 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "cff421422e984847a6be313bda770b46"
format_stamp: "Formatted at 2026-08-10 18:58:54 on dist-test-slave-znh6"
server_key: "10a2413550c40c9a94effdd9094cf41a"
server_key_iv: "4fee1a1477bc2a61a8451be1364fef9f"
server_key_version: "encryptionkey@0"
I20260810 18:58:54.549391 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "cff421422e984847a6be313bda770b46"
format_stamp: "Formatted at 2026-08-10 18:58:54 on dist-test-slave-znh6"
server_key: "10a2413550c40c9a94effdd9094cf41a"
server_key_iv: "4fee1a1477bc2a61a8451be1364fef9f"
server_key_version: "encryptionkey@0"
I20260810 18:58:54.553195 4003 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:58:54.555696 4030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:54.556895 4003 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:54.557039 4003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "cff421422e984847a6be313bda770b46"
format_stamp: "Formatted at 2026-08-10 18:58:54 on dist-test-slave-znh6"
server_key: "10a2413550c40c9a94effdd9094cf41a"
server_key_iv: "4fee1a1477bc2a61a8451be1364fef9f"
server_key_version: "encryptionkey@0"
I20260810 18:58:54.557155 4003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:54.578047 4003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:54.578790 4003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:54.579046 4003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:54.586877 4003 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:36763
I20260810 18:58:54.586902 4103 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:36763 every 8 connection(s)
I20260810 18:58:54.587880 4003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:54.590569 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 4003
I20260810 18:58:54.590699 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:54.590969 4104 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:54.591074 28238 external_mini_cluster.cc:1428] Setting key 3a886b1f7aee26b0bec5d7f32366de30
I20260810 18:58:54.597396 4104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46: Bootstrap starting.
I20260810 18:58:54.599956 4104 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:54.600236 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:54.592473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43122 (local address 127.27.147.190:36763)
0810 18:58:54.593128 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:54.593144 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:54.593180 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:58:54.593806 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:54.593824 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:54.594227 (+ 403us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:54.594567 (+ 340us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:54.595049 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.596029 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.596894 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.597359 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.597552 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.598430 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:54.598435 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:54.598455 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:58:54.598898 (+ 443us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:54.598965 (+ 67us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:58:54.598987 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:54.599096 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:54.599224 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 18:58:54.599496 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":145,"threads_started":1}
I20260810 18:58:54.600867 4104 log.cc:824] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:54.603148 4104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46: No bootstrap required, opened a new log
I20260810 18:58:54.603226 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 18:58:54.606842 4104 raft_consensus.cc:348] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff421422e984847a6be313bda770b46" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36763 } }
I20260810 18:58:54.607088 4104 raft_consensus.cc:374] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:54.607188 4104 raft_consensus.cc:729] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cff421422e984847a6be313bda770b46, State: Initialized, Role: FOLLOWER
I20260810 18:58:54.607668 4104 consensus_queue.cc:260] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff421422e984847a6be313bda770b46" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36763 } }
I20260810 18:58:54.607824 4104 raft_consensus.cc:388] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:54.607913 4104 raft_consensus.cc:482] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:54.608014 4104 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:54.608899 4104 raft_consensus.cc:504] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff421422e984847a6be313bda770b46" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36763 } }
I20260810 18:58:54.609241 4104 leader_election.cc:302] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cff421422e984847a6be313bda770b46; no voters:
I20260810 18:58:54.609571 4104 leader_election.cc:288] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:54.609663 4114 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:54.609969 4114 raft_consensus.cc:686] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [term 1 LEADER]: Becoming Leader. State: Replica: cff421422e984847a6be313bda770b46, State: Running, Role: LEADER
I20260810 18:58:54.610268 4114 consensus_queue.cc:237] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff421422e984847a6be313bda770b46" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36763 } }
I20260810 18:58:54.610885 4104 sys_catalog.cc:564] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:54.612056 4116 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cff421422e984847a6be313bda770b46. Latest consensus state: current_term: 1 leader_uuid: "cff421422e984847a6be313bda770b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff421422e984847a6be313bda770b46" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36763 } } }
I20260810 18:58:54.612036 4115 sys_catalog.cc:455] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cff421422e984847a6be313bda770b46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cff421422e984847a6be313bda770b46" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 36763 } } }
I20260810 18:58:54.612222 4116 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:54.612283 4115 sys_catalog.cc:458] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:54.612680 4134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:54.615435 4134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:54.621516 4134 catalog_manager.cc:1347] Generated new cluster ID: 6acedd2247d84b8d94dc5df9e16c949b
I20260810 18:58:54.621582 4134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:54.636430 4134 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:54.637697 4134 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:54.659828 4134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cff421422e984847a6be313bda770b46: Generated new TSK 0
I20260810 18:58:54.660748 4134 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:58:55.130630 4107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:54.675890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43126 (local address 127.27.147.190:36763)
0810 18:58:54.676034 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:54.676039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:54.676206 (+ 167us) server_negotiation.cc:407] Connection header received
0810 18:58:54.676377 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:54.676380 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:54.676437 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:54.676571 (+ 134us) server_negotiation.cc:226] Negotiated authn=JWT
0810 18:58:54.676774 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.677621 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.678496 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.678891 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.679002 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.679282 (+ 280us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 18:58:55.130214 (+450932us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:58:55.130503 (+ 289us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:43126: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":55}
W20260810 18:58:55.130707 4107 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:43126: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 18:58:55.131181 4162 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:54.675807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:36763 (local address 127.0.0.1:43126)
0810 18:58:54.676118 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:54.676132 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:54.676229 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:54.676564 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:54.676566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:54.676573 (+ 7us) client_negotiation.cc:189] Negotiated authn=JWT
0810 18:58:54.676678 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:54.676682 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.677751 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.677754 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:54.678379 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:54.678387 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:54.679016 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:54.679019 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:54.679086 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:54.679108 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 18:58:55.130470 (+451362us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 18:58:55.130663 (+ 193us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.27.147.190:36763: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":217,"thread_start_us":129,"threads_started":1}
I20260810 18:58:55.131835 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 4003
2026-08-10T18:58:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (767 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (128 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (127 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:55 dist-test-slave-znh6 krb5kdc[4186](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:55 dist-test-slave-znh6 krb5kdc[4186](info): set up 2 sockets
Aug 10 18:58:55 dist-test-slave-znh6 krb5kdc[4186](info): commencing operation
krb5kdc: starting...
W20260810 18:58:56.634403 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.218s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:56 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:58:56.763949 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:46109
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:46109
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:56.876569 4205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:56.876819 4205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:56.876859 4205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:56.880935 4205 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:56.881004 4205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:56.881023 4205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:56.881040 4205 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:56.881078 4205 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:56.886268 4205 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:46109
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:46109
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:56.887347 4205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:56.888366 4205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:56.897570 4211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:56.897584 4212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:56.897660 4215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:56.897603 4205 server_base.cc:1034] running on GCE node
I20260810 18:58:56.898391 4205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:56.899511 4205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:56.900671 4205 hybrid_clock.cc:648] HybridClock initialized: now 1786388336900640 us; error 43 us; skew 500 ppm
Aug 10 18:58:56 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388336, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:56.903366 4205 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:58:56.904264 4205 webserver.cc:460] Webserver started at http://127.27.147.190:42197/ using document root <none> and password file <none>
I20260810 18:58:56.904825 4205 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:56.904912 4205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:56.905151 4205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:56.907375 4205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "4e56a8db31034010b3f499991fae7a17"
format_stamp: "Formatted at 2026-08-10 18:58:56 on dist-test-slave-znh6"
server_key: "79a410ce01c67dbce4ca8f576d13eb1f"
server_key_iv: "ba4ce2a1740c04ea468244d0c6c528ae"
server_key_version: "encryptionkey@0"
I20260810 18:58:56.907979 4205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "4e56a8db31034010b3f499991fae7a17"
format_stamp: "Formatted at 2026-08-10 18:58:56 on dist-test-slave-znh6"
server_key: "79a410ce01c67dbce4ca8f576d13eb1f"
server_key_iv: "ba4ce2a1740c04ea468244d0c6c528ae"
server_key_version: "encryptionkey@0"
I20260810 18:58:56.911821 4205 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 18:58:56.914220 4224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:56.915531 4205 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:58:56.915681 4205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "4e56a8db31034010b3f499991fae7a17"
format_stamp: "Formatted at 2026-08-10 18:58:56 on dist-test-slave-znh6"
server_key: "79a410ce01c67dbce4ca8f576d13eb1f"
server_key_iv: "ba4ce2a1740c04ea468244d0c6c528ae"
server_key_version: "encryptionkey@0"
I20260810 18:58:56.915808 4205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:56.948359 4205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:56.951653 4205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:56.951866 4205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:56.959331 4205 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:46109
I20260810 18:58:56.959352 4301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:46109 every 8 connection(s)
I20260810 18:58:56.960309 4205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:56.960821 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 4205
I20260810 18:58:56.960953 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:56.961230 28238 external_mini_cluster.cc:1428] Setting key 538e3ae42bec5796cee0a57d4739c135
I20260810 18:58:56.963702 4302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 18:58:56 dist-test-slave-znh6 krb5kdc[4186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388336, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:56.970691 4302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17: Bootstrap starting.
I20260810 18:58:56.973335 4302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:56.974259 4302 log.cc:824] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:56.976063 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:56.962653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38548 (local address 127.27.147.190:46109)
0810 18:58:56.963116 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:56.963135 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:56.963177 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:58:56.964061 (+ 884us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:56.964084 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:56.964492 (+ 408us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:56.964864 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:56.965527 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:56.967043 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:56.967883 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:56.968549 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:56.968723 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:56.971380 (+ 2657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:56.971386 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:56.971399 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:56.971427 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:56.973086 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:56.973610 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:56.973615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:56.973620 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:56.973735 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:56.974034 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:56.974037 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:56.974038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:56.974497 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:56.974654 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:56.975060 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 18:58:56.975343 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":164,"threads_started":1}
I20260810 18:58:56.976545 4302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17: No bootstrap required, opened a new log
I20260810 18:58:56.980756 4302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e56a8db31034010b3f499991fae7a17" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 46109 } }
I20260810 18:58:56.980994 4302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:56.981072 4302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e56a8db31034010b3f499991fae7a17, State: Initialized, Role: FOLLOWER
I20260810 18:58:56.981528 4302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e56a8db31034010b3f499991fae7a17" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 46109 } }
I20260810 18:58:56.981679 4302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:56.981763 4302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:56.981866 4302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:56.982898 4302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e56a8db31034010b3f499991fae7a17" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 46109 } }
I20260810 18:58:56.983278 4302 leader_election.cc:302] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e56a8db31034010b3f499991fae7a17; no voters:
I20260810 18:58:56.983623 4302 leader_election.cc:288] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:56.983732 4311 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:56.984040 4311 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [term 1 LEADER]: Becoming Leader. State: Replica: 4e56a8db31034010b3f499991fae7a17, State: Running, Role: LEADER
I20260810 18:58:56.984362 4311 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e56a8db31034010b3f499991fae7a17" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 46109 } }
I20260810 18:58:56.985002 4302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:56.987210 4312 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e56a8db31034010b3f499991fae7a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e56a8db31034010b3f499991fae7a17" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 46109 } } }
I20260810 18:58:56.987354 4312 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:56.987607 4313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e56a8db31034010b3f499991fae7a17. Latest consensus state: current_term: 1 leader_uuid: "4e56a8db31034010b3f499991fae7a17" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e56a8db31034010b3f499991fae7a17" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 46109 } } }
I20260810 18:58:56.987797 4313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17 [sys.catalog]: This master's current role is: LEADER
W20260810 18:58:56.989748 4336 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:58:56.989833 4336 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:58:56.989912 4337 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:56.990608 4337 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:56.995523 4337 catalog_manager.cc:1347] Generated new cluster ID: b997c2d24870473d8039431f81705666
I20260810 18:58:56.995585 4337 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:57.029613 4337 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:57.030587 4337 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:57.044638 4337 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e56a8db31034010b3f499991fae7a17: Generated new TSK 0
I20260810 18:58:57.045372 4337 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:58:57.078065 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:46109
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:57.192688 4342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:57.192972 4342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:57.193044 4342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:57.197008 4342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:57.197088 4342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:57.197175 4342 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:57.202528 4342 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4342
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:57.203632 4342 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:57.204635 4342 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:57.214807 4350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:57.214805 4351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:57.214888 4342 server_base.cc:1034] running on GCE node
W20260810 18:58:57.214843 4354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:57.215472 4342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:57.216138 4342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:57.217329 4342 hybrid_clock.cc:648] HybridClock initialized: now 1786388337217310 us; error 24 us; skew 500 ppm
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.220357 4342 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:58:57.221329 4342 webserver.cc:460] Webserver started at http://127.27.147.129:41775/ using document root <none> and password file <none>
I20260810 18:58:57.221899 4342 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:57.221964 4342 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:57.222194 4342 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:57.224320 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "ad5124d1658741baaf7bd3063d2f6836"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "dfd5f0a6b0ed96e6fdf2ce6d3e94947f"
server_key_iv: "ac86c8fc02741f13f1fbb14631daa8ae"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.224898 4342 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "ad5124d1658741baaf7bd3063d2f6836"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "dfd5f0a6b0ed96e6fdf2ce6d3e94947f"
server_key_iv: "ac86c8fc02741f13f1fbb14631daa8ae"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.228693 4342 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:58:57.231123 4366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.232301 4342 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 18:58:57.232430 4342 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "ad5124d1658741baaf7bd3063d2f6836"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "dfd5f0a6b0ed96e6fdf2ce6d3e94947f"
server_key_iv: "ac86c8fc02741f13f1fbb14631daa8ae"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.232528 4342 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:57.264842 4342 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:57.268162 4342 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:57.268376 4342 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:57.269007 4342 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:57.269850 4342 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:57.269898 4342 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.269968 4342 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:57.269999 4342 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.278927 4342 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:38423
I20260810 18:58:57.278931 4529 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:38423 every 8 connection(s)
I20260810 18:58:57.279851 4342 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:57.285306 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 4342
I20260810 18:58:57.285470 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:57.285799 28238 external_mini_cluster.cc:1428] Setting key f5ffda8c9ac7bcccd7d8e44714bebe55
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:57.295125 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.281689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:55015 (local address 127.27.147.190:46109)
0810 18:58:57.281844 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.281849 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.282716 (+ 867us) server_negotiation.cc:407] Connection header received
0810 18:58:57.284128 (+ 1412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.284131 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.284177 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.284251 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.285488 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.286279 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.287627 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.287856 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.287948 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.291158 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:57.291161 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:57.291163 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:57.291186 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:57.292707 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.293337 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.293339 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.293341 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.293418 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.293844 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.293847 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.293849 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.294056 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:57.294135 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:57.294882 (+ 747us) server_negotiation.cc:299] Negotiation successful
0810 18:58:57.294998 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:58:57.296307 4532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.282099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:46109 (local address 127.27.147.129:55015)
0810 18:58:57.282560 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:57.282603 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:57.283932 (+ 1329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:57.284413 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:57.284421 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:57.284805 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:57.285284 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:57.285297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.286424 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.286427 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:57.287508 (+ 1081us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:57.287520 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.288020 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.288026 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:57.288255 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.288881 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:57.288906 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:57.290989 (+ 2083us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:57.292828 (+ 1839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:57.292836 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:57.292853 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:57.293231 (+ 378us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:57.293507 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:57.293512 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:57.293516 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:57.293721 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:57.294170 (+ 449us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:57.294179 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:57.294549 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 18:58:57.294767 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 18:58:57.295169 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":116,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:57.297603 4530 heartbeater.cc:347] Connected to a master server at 127.27.147.190:46109
I20260810 18:58:57.297849 4530 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:57.298439 4530 heartbeater.cc:511] Master 127.27.147.190:46109 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 18:58:57.300135 4252 ts_manager.cc:194] Registered new tserver with Master: ad5124d1658741baaf7bd3063d2f6836 (127.27.147.129:38423)
I20260810 18:58:57.301303 4252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:55015
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:58:57.317754 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:46109
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:57.431634 4537 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:57.431869 4537 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:57.431910 4537 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:57.435925 4537 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:57.435979 4537 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:57.436034 4537 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:57.441018 4537 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4537
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:57.441922 4537 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:57.442871 4537 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:58:57.452349 4537 server_base.cc:1034] running on GCE node
W20260810 18:58:57.452342 4550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:57.452366 4547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:57.452616 4553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:57.453007 4537 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:57.453608 4537 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:57.454887 4537 hybrid_clock.cc:648] HybridClock initialized: now 1786388337454864 us; error 33 us; skew 500 ppm
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.457610 4537 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:58:57.458616 4537 webserver.cc:460] Webserver started at http://127.27.147.130:37769/ using document root <none> and password file <none>
I20260810 18:58:57.459323 4537 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:57.459386 4537 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:57.459615 4537 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:57.462059 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "83711d99e8d3413ba15f81cf11a56f2d"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "aa98bb55420a6276d1d3a9938edfc787"
server_key_iv: "66c9013ba316e47aec36be0392b0b249"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.462644 4537 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "83711d99e8d3413ba15f81cf11a56f2d"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "aa98bb55420a6276d1d3a9938edfc787"
server_key_iv: "66c9013ba316e47aec36be0392b0b249"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.466228 4537 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:58:57.468604 4565 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.469790 4537 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:58:57.469918 4537 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "83711d99e8d3413ba15f81cf11a56f2d"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "aa98bb55420a6276d1d3a9938edfc787"
server_key_iv: "66c9013ba316e47aec36be0392b0b249"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.470017 4537 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:57.484813 4537 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:57.488193 4537 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:57.488394 4537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:57.488994 4537 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:57.489818 4537 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:57.489866 4537 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.489938 4537 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:57.489974 4537 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.498843 4537 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:34949
I20260810 18:58:57.498895 4727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:34949 every 8 connection(s)
I20260810 18:58:57.499749 4537 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:57.504773 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 4537
I20260810 18:58:57.504905 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:58:57.505184 28238 external_mini_cluster.cc:1428] Setting key 80b2917f6820485cfbf983b9a4f5edad
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:57.514212 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.501550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:48561 (local address 127.27.147.190:46109)
0810 18:58:57.501694 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.501699 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.502554 (+ 855us) server_negotiation.cc:407] Connection header received
0810 18:58:57.503980 (+ 1426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.503983 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.504032 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.504110 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.505384 (+ 1274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.506005 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.507436 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.507834 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.507975 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.510316 (+ 2341us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:57.510323 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:57.510329 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:57.510362 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:57.512088 (+ 1726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.512580 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.512584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.512586 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.512684 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.513012 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.513018 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.513022 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.513292 (+ 270us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:57.513412 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:57.513858 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 18:58:57.514018 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:58:57.514950 4731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.501968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:46109 (local address 127.27.147.130:48561)
0810 18:58:57.502400 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:57.502449 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:57.503776 (+ 1327us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:57.504267 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:57.504273 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:57.504663 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:57.505203 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:57.505217 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.506130 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.506134 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:57.507299 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:57.507309 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.507995 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.507998 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:57.508135 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.508715 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:57.508731 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:57.510152 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:57.512203 (+ 2051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:57.512207 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:57.512219 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:57.512475 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:57.512784 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:57.512786 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:57.512787 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:57.512912 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:57.513398 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:57.513403 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:57.513623 (+ 220us) client_negotiation.cc:769] Sending connection context
0810 18:58:57.513773 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 18:58:57.514012 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":106,"threads_started":1}
I20260810 18:58:57.516201 4728 heartbeater.cc:347] Connected to a master server at 127.27.147.190:46109
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:58:57.516484 4728 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:57.516970 4728 heartbeater.cc:511] Master 127.27.147.190:46109 requested a full tablet report, sending...
I20260810 18:58:57.518077 4252 ts_manager.cc:194] Registered new tserver with Master: 83711d99e8d3413ba15f81cf11a56f2d (127.27.147.130:34949)
I20260810 18:58:57.518695 4252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:48561
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:58:57.533799 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:46109
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:33065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:57.646332 4737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:57.646621 4737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:57.646694 4737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:57.650804 4737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:57.650898 4737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:57.650987 4737 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:58:57.656064 4737 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:33065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:46109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:57.657079 4737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:57.658092 4737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:57.668496 4745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:57.668552 4737 server_base.cc:1034] running on GCE node
W20260810 18:58:57.668563 4744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:57.668489 4754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:57.669130 4737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:57.669752 4737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:57.670903 4737 hybrid_clock.cc:648] HybridClock initialized: now 1786388337670889 us; error 27 us; skew 500 ppm
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.673542 4737 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:58:57.674532 4737 webserver.cc:460] Webserver started at http://127.27.147.131:44869/ using document root <none> and password file <none>
I20260810 18:58:57.675185 4737 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:57.675274 4737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:57.675489 4737 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:57.677500 4737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "8597f01904ab4537aee83f3f4e37aa1a"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "c3e7c21e54d843e21e83d139f241f672"
server_key_iv: "6ec526389bbb05f336aa76c4c3ef060f"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.678050 4737 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "8597f01904ab4537aee83f3f4e37aa1a"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "c3e7c21e54d843e21e83d139f241f672"
server_key_iv: "6ec526389bbb05f336aa76c4c3ef060f"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.681813 4737 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.006s
I20260810 18:58:57.684223 4761 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.685463 4737 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:57.685596 4737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "8597f01904ab4537aee83f3f4e37aa1a"
format_stamp: "Formatted at 2026-08-10 18:58:57 on dist-test-slave-znh6"
server_key: "c3e7c21e54d843e21e83d139f241f672"
server_key_iv: "6ec526389bbb05f336aa76c4c3ef060f"
server_key_version: "encryptionkey@0"
I20260810 18:58:57.685702 4737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:57.708437 4737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:57.711287 4737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:57.711473 4737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:57.712037 4737 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:57.712829 4737 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:57.712898 4737 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.712975 4737 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:57.713014 4737 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:57.722373 4737 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:43613
I20260810 18:58:57.722405 4925 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:43613 every 8 connection(s)
I20260810 18:58:57.723270 4737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:58:57.730086 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 4737
I20260810 18:58:57.730225 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:58:57.730505 28238 external_mini_cluster.cc:1428] Setting key e9cde8347ef269c834a9fb13d86bdc58
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:58:57.736346 4307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.724961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:40065 (local address 127.27.147.190:46109)
0810 18:58:57.725118 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.725123 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.725917 (+ 794us) server_negotiation.cc:407] Connection header received
0810 18:58:57.727331 (+ 1414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.727334 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.727384 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.727448 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.728530 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.729059 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.730186 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.730454 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.730567 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.733304 (+ 2737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:57.733307 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:57.733309 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:57.733333 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:57.734490 (+ 1157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.735001 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.735004 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.735005 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.735085 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.735376 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.735379 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.735380 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.735550 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:57.735622 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:57.736037 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 18:58:57.736135 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:58:57.737066 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.725340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:46109 (local address 127.27.147.131:40065)
0810 18:58:57.725773 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:57.725814 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:57.727128 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:57.727610 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:57.727616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:57.727949 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:57.728396 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:57.728408 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.729159 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.729162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:57.730053 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:57.730062 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.730605 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.730610 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:57.730798 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.731472 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:57.731495 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:57.733161 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:57.734589 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:57.734594 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:57.734605 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:57.734913 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:57.735165 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:57.735167 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:57.735168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:57.735294 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:57.735631 (+ 337us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:57.735636 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:57.735838 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 18:58:57.735986 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 18:58:57.736240 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":117,"threads_started":1}
I20260810 18:58:57.738196 4928 heartbeater.cc:347] Connected to a master server at 127.27.147.190:46109
I20260810 18:58:57.738394 4928 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:57.738870 4928 heartbeater.cc:511] Master 127.27.147.190:46109 requested a full tablet report, sending...
I20260810 18:58:57.739849 4252 ts_manager.cc:194] Registered new tserver with Master: 8597f01904ab4537aee83f3f4e37aa1a (127.27.147.131:43613)
I20260810 18:58:57.740398 4252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:40065
I20260810 18:58:57.744239 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/test-admin.keytab.
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.760850 28238 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 18:58:57.763535 4939 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:57.762150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39066 (local address 127.27.147.129:38423)
0810 18:58:57.762425 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.762445 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.762469 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:58:57.762520 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.762525 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.762703 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.762867 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.763310 (+ 443us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39066: BlockingRecv error: recv got EOF from 127.0.0.1:39066 (error 108)
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
W20260810 18:58:57.763592 28238 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.764599 28238 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.765605 28238 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:57.775178 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.766463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39072 (local address 127.27.147.129:38423)
0810 18:58:57.766613 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.766617 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.766711 (+ 94us) server_negotiation.cc:407] Connection header received
0810 18:58:57.766944 (+ 233us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.766951 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.766991 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.767074 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.767617 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.768405 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.769243 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.769610 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.769724 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.771969 (+ 2245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:57.771978 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:57.771991 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:57.772014 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:57.773485 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.773868 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.773873 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.773874 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.773979 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.774220 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.774223 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.774224 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.774472 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:57.774624 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:57.774798 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 18:58:57.775034 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20260810 18:58:57.778803 4939 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:57.777794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.27.147.129:38423)
0810 18:58:57.777904 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.777908 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.778045 (+ 137us) server_negotiation.cc:407] Connection header received
0810 18:58:57.778192 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.778195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.778228 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.778304 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.778707 (+ 403us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39080: BlockingRecv error: recv got EOF from 127.0.0.1:39080 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 18:58:57.779147 28238 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc)
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.780105 28238 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.781090 28238 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:57.789831 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.781856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39082 (local address 127.27.147.129:38423)
0810 18:58:57.781976 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.781981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.782097 (+ 116us) server_negotiation.cc:407] Connection header received
0810 18:58:57.782231 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.782234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.782271 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.782357 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.782891 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.783398 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.784172 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.784413 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.784516 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.786854 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:57.786858 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:57.786861 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:57.786886 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:57.788428 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.788826 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.788828 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.788829 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.788895 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.789191 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.789195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.789197 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.789351 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:57.789444 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:57.789641 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 18:58:57.789729 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 18:58:57.792358 4939 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:58:57.791397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39094 (local address 127.27.147.129:38423)
0810 18:58:57.791535 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.791539 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.791652 (+ 113us) server_negotiation.cc:407] Connection header received
0810 18:58:57.791763 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.791768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.791807 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.791891 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.792271 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39094: BlockingRecv error: recv got EOF from 127.0.0.1:39094 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 18:58:57.792500 28238 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786388305279399-28238-0/krb5kdc/krb5cc' of type FILE
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.793407 28238 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:57.794267 28238 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4186](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 18:58:57.802142 4939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:57.795007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39110 (local address 127.27.147.129:38423)
0810 18:58:57.795112 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:57.795115 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:57.795244 (+ 129us) server_negotiation.cc:407] Connection header received
0810 18:58:57.795317 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:57.795319 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:57.795349 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:57.795412 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:57.795837 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.796301 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.797098 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:57.797293 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:57.797364 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:57.799319 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:57.799322 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:57.799329 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:57.799351 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:57.800643 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.801045 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.801050 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.801052 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.801126 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:57.801393 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:57.801397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:57.801399 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:57.801563 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:57.801685 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:57.801911 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 18:58:57.802026 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 18:58:57.803035 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 4342
I20260810 18:58:57.819479 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 4537
I20260810 18:58:57.832302 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 4737
I20260810 18:58:57.843609 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 4205
2026-08-10T18:58:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (2458 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4957](info): set up 2 sockets
Aug 10 18:58:57 dist-test-slave-znh6 krb5kdc[4957](info): commencing operation
krb5kdc: starting...
W20260810 18:58:59.090436 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:58:59 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:58:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:58:59Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab.
Entry for principal oryx/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab.
I20260810 18:58:59.216109 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:42793
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab
--principal=oryx/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:42793
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:59.326371 4975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:59.326663 4975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:59.326735 4975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:59.330613 4975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:58:59.330696 4975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:59.330735 4975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:59.330789 4975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:58:59.330842 4975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:58:59.335968 4975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:42793
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab
--principal=oryx/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:42793
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:59.336974 4975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:59.337957 4975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:59.346431 4984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:59.346441 4982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:59.346443 4986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:59.346789 4975 server_base.cc:1034] running on GCE node
I20260810 18:58:59.347384 4975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:59.348435 4975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:59.349624 4975 hybrid_clock.cc:648] HybridClock initialized: now 1786388339349605 us; error 35 us; skew 500 ppm
Aug 10 18:58:59 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:59.355368 4975 init.cc:377] Logged in from keytab as oryx/127.27.147.190@KRBTEST.COM (short username oryx)
I20260810 18:58:59.356210 4975 webserver.cc:460] Webserver started at http://127.27.147.190:33709/ using document root <none> and password file <none>
I20260810 18:58:59.356729 4975 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:59.356787 4975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:59.356949 4975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:59.359042 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "b007540de45341a0a6c5ef083c571502"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "39016123a4a4ca362551ca8c6e93ef0c"
server_key_iv: "a9eecfea086c97e503108330053dcc1a"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.359637 4975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "b007540de45341a0a6c5ef083c571502"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "39016123a4a4ca362551ca8c6e93ef0c"
server_key_iv: "a9eecfea086c97e503108330053dcc1a"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.363380 4975 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 18:58:59.365622 5001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:59.366803 4975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:59.366962 4975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "b007540de45341a0a6c5ef083c571502"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "39016123a4a4ca362551ca8c6e93ef0c"
server_key_iv: "a9eecfea086c97e503108330053dcc1a"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.367054 4975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:59.379952 4975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:59.382555 4975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:59.382700 4975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:59.389719 4975 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:42793
I20260810 18:58:59.389991 5076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:42793 every 8 connection(s)
I20260810 18:58:59.390697 4975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:58:59.391921 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 4975
I20260810 18:58:59.392071 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:58:59.392338 28238 external_mini_cluster.cc:1428] Setting key 132b4b098e8ee01c0f7be0a644b9c526
I20260810 18:58:59.393828 5078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:58:59.400426 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: Bootstrap starting.
Aug 10 18:58:59 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:58:59.402748 5078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: Neither blocks nor log segments found. Creating new log.
I20260810 18:58:59.403625 5078 log.cc:824] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: Log is configured to *not* fsync() on all Append() calls
I20260810 18:58:59.405800 5078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: No bootstrap required, opened a new log
I20260810 18:58:59.406745 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:59.393758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.27.147.190:42793)
0810 18:58:59.394196 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:59.394217 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:59.394259 (+ 42us) server_negotiation.cc:407] Connection header received
0810 18:58:59.395117 (+ 858us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:59.395140 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:59.395545 (+ 405us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:59.395893 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:59.396568 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.398069 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.398996 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.399459 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.399635 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:59.402143 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:59.402149 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:59.402162 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:59.402190 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:59.404010 (+ 1820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:59.404460 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:59.404470 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:59.404475 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:59.404585 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:59.404884 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:59.404886 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:59.404887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:59.405254 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:59.405404 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:59.405716 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 18:58:59.406031 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":138,"threads_started":1}
I20260810 18:58:59.408759 5078 raft_consensus.cc:348] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } }
I20260810 18:58:59.408995 5078 raft_consensus.cc:374] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:58:59.409061 5078 raft_consensus.cc:729] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b007540de45341a0a6c5ef083c571502, State: Initialized, Role: FOLLOWER
I20260810 18:58:59.409526 5078 consensus_queue.cc:260] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } }
I20260810 18:58:59.409653 5078 raft_consensus.cc:388] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:58:59.409699 5078 raft_consensus.cc:482] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:58:59.409770 5078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:58:59.410588 5078 raft_consensus.cc:504] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } }
I20260810 18:58:59.410929 5078 leader_election.cc:302] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b007540de45341a0a6c5ef083c571502; no voters:
I20260810 18:58:59.411190 5078 leader_election.cc:288] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:58:59.411309 5085 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:58:59.411557 5085 raft_consensus.cc:686] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 1 LEADER]: Becoming Leader. State: Replica: b007540de45341a0a6c5ef083c571502, State: Running, Role: LEADER
I20260810 18:58:59.411891 5085 consensus_queue.cc:237] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } }
I20260810 18:58:59.412379 5078 sys_catalog.cc:564] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:58:59.413707 5087 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b007540de45341a0a6c5ef083c571502. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } }
I20260810 18:58:59.413836 5087 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:59.414114 5086 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } }
I20260810 18:58:59.414247 5086 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:58:59.414604 5102 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:58:59.417444 5102 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:58:59.422488 5102 catalog_manager.cc:1347] Generated new cluster ID: e659b0ff0a5c428caa87ff428685bbf3
I20260810 18:58:59.422552 5102 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:58:59.444048 5102 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:58:59.444985 5102 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:58:59.455729 5102 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: Generated new TSK 0
I20260810 18:58:59.456643 5102 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab.
Entry for principal oryx/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab.
I20260810 18:58:59.489562 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42793
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab
--principal=oryx/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:59.602985 5120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:59.603227 5120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:59.603286 5120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:59.607259 5120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:59.607311 5120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:59.607367 5120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:58:59.612367 5120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab
--principal=oryx/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:59.613445 5120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:59.614482 5120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:59.624122 5129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:59.624104 5132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:59.624102 5130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:59.624509 5120 server_base.cc:1034] running on GCE node
I20260810 18:58:59.624854 5120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:59.625391 5120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:59.626541 5120 hybrid_clock.cc:648] HybridClock initialized: now 1786388339626514 us; error 41 us; skew 500 ppm
Aug 10 18:58:59 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:59.629305 5120 init.cc:377] Logged in from keytab as oryx/127.27.147.129@KRBTEST.COM (short username oryx)
I20260810 18:58:59.630172 5120 webserver.cc:460] Webserver started at http://127.27.147.129:38713/ using document root <none> and password file <none>
I20260810 18:58:59.630690 5120 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:59.630750 5120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:59.630990 5120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:59.632973 5120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "f34121eec10941dfbb6f650604c8656f"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "b5dfa05c06fd46b017892b51ab98c3ad"
server_key_iv: "943906e93ce5bc25272053bccd0f589d"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.633505 5120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "f34121eec10941dfbb6f650604c8656f"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "b5dfa05c06fd46b017892b51ab98c3ad"
server_key_iv: "943906e93ce5bc25272053bccd0f589d"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.637697 5120 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:58:59.640035 5145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:59.641203 5120 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:58:59.641299 5120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "f34121eec10941dfbb6f650604c8656f"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "b5dfa05c06fd46b017892b51ab98c3ad"
server_key_iv: "943906e93ce5bc25272053bccd0f589d"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.641381 5120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:59.660043 5120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:59.663177 5120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:59.663339 5120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:59.663868 5120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:59.664649 5120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:59.664705 5120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:59.664745 5120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:59.664767 5120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:59.673666 5120 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:37149
I20260810 18:58:59.673700 5319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:37149 every 8 connection(s)
I20260810 18:58:59.674541 5120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:58:59.675972 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 5120
I20260810 18:58:59.676103 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:58:59.676429 28238 external_mini_cluster.cc:1428] Setting key 9ff58a762cd76c9a3da3017b81b2e987
Aug 10 18:58:59 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for oryx/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:58:59.689448 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:59.676396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:39065 (local address 127.27.147.190:42793)
0810 18:58:59.676537 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:59.676542 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:59.677479 (+ 937us) server_negotiation.cc:407] Connection header received
0810 18:58:59.678948 (+ 1469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:59.678952 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:59.679005 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:59.679082 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:59.680395 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.680936 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.682675 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.682956 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.683036 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:59.685834 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:59.685837 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:59.685839 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:59.685863 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:59.687323 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:59.687824 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:59.687829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:59.687831 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:59.687913 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:59.688293 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:59.688298 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:59.688299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:59.688482 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:59.688584 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:59.689113 (+ 529us) server_negotiation.cc:299] Negotiation successful
0810 18:58:59.689280 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.130@KRBTEST.COM" created.
I20260810 18:58:59.690296 5324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:59.676893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:39065)
0810 18:58:59.677347 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:59.677388 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:59.678702 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:59.679242 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:59.679248 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:59.679584 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:59.680218 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:59.680239 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.681060 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.681065 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:59.682544 (+ 1479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:59.682559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.683110 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.683116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:59.683361 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:59.683940 (+ 579us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:59.683975 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:59.685660 (+ 1685us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:59.687442 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:59.687447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:59.687462 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:59.687719 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:59.688020 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:59.688023 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:59.688025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:59.688195 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:59.688599 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:59.688607 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:59.688843 (+ 236us) client_negotiation.cc:769] Sending connection context
0810 18:58:59.689027 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 18:58:59.689342 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":270,"thread_start_us":108,"threads_started":1}
I20260810 18:58:59.691613 5320 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42793
I20260810 18:58:59.691892 5320 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:59.692487 5320 heartbeater.cc:511] Master 127.27.147.190:42793 requested a full tablet report, sending...
I20260810 18:58:59.694096 5024 ts_manager.cc:194] Registered new tserver with Master: f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
I20260810 18:58:59.695329 5024 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:39065
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab.
Entry for principal oryx/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab.
I20260810 18:58:59.705745 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42793
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab
--principal=oryx/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:58:59.817190 5328 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:58:59.817476 5328 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:58:59.817553 5328 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:58:59.821781 5328 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:58:59.821856 5328 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:58:59.821935 5328 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:58:59.827140 5328 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab
--principal=oryx/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5328
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:58:59.828147 5328 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:58:59.829166 5328 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:58:59.838936 5338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:59.839041 5328 server_base.cc:1034] running on GCE node
W20260810 18:58:59.838974 5335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:58:59.838936 5336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:58:59.839637 5328 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:58:59.840229 5328 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:58:59.841413 5328 hybrid_clock.cc:648] HybridClock initialized: now 1786388339841392 us; error 30 us; skew 500 ppm
Aug 10 18:58:59 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:58:59.844172 5328 init.cc:377] Logged in from keytab as oryx/127.27.147.130@KRBTEST.COM (short username oryx)
I20260810 18:58:59.845124 5328 webserver.cc:460] Webserver started at http://127.27.147.130:46431/ using document root <none> and password file <none>
I20260810 18:58:59.845752 5328 fs_manager.cc:359] Metadata directory not provided
I20260810 18:58:59.845816 5328 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:58:59.846050 5328 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:58:59.848110 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "c6ab1c37242045b0806525c8ecbb76c3"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "aeca7b11c87fa7c5d0eb125b2cc2daee"
server_key_iv: "c2145e349b125b4e823923df26fab203"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.848676 5328 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "c6ab1c37242045b0806525c8ecbb76c3"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "aeca7b11c87fa7c5d0eb125b2cc2daee"
server_key_iv: "c2145e349b125b4e823923df26fab203"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.852276 5328 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:58:59.854573 5346 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:59.855770 5328 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:58:59.855890 5328 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "c6ab1c37242045b0806525c8ecbb76c3"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "aeca7b11c87fa7c5d0eb125b2cc2daee"
server_key_iv: "c2145e349b125b4e823923df26fab203"
server_key_version: "encryptionkey@0"
I20260810 18:58:59.855990 5328 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:58:59.878947 5328 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:58:59.881729 5328 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:58:59.881933 5328 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:58:59.882534 5328 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:58:59.883394 5328 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:58:59.883442 5328 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:59.883513 5328 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:58:59.883553 5328 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:58:59.892307 5328 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:44075
I20260810 18:58:59.892339 5505 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:44075 every 8 connection(s)
I20260810 18:58:59.893175 5328 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:58:59.902055 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 5328
I20260810 18:58:59.902202 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
Aug 10 18:58:59 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:58:59.902539 28238 external_mini_cluster.cc:1428] Setting key 84e0513be2558deffac1387106e8f0c4
I20260810 18:58:59.906754 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:59.894958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:41353 (local address 127.27.147.190:42793)
0810 18:58:59.895111 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 18:58:59.895115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:58:59.895930 (+ 815us) server_negotiation.cc:407] Connection header received
0810 18:58:59.897266 (+ 1336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:58:59.897269 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:58:59.897316 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:58:59.897392 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:58:59.898550 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.899095 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.900163 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.900369 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.900442 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:59.902940 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:58:59.902943 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:58:59.902946 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:58:59.902979 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:58:59.904365 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:59.905036 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:59.905039 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:59.905040 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:59.905111 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:58:59.905496 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:58:59.905499 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:58:59.905500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:58:59.905665 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:58:59.905740 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:58:59.906446 (+ 706us) server_negotiation.cc:299] Negotiation successful
0810 18:58:59.906583 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 18:58:59.907554 5510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:58:59.895342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:41353)
0810 18:58:59.895780 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 18:58:59.895822 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:58:59.897067 (+ 1245us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:58:59.897559 (+ 492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:58:59.897566 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:58:59.897924 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:58:59.898416 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:59.898430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.899195 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.899197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:59.900063 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:58:59.900073 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:58:59.900489 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:58:59.900492 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:58:59.900625 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:58:59.901179 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:58:59.901195 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:58:59.902754 (+ 1559us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:58:59.904519 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:59.904527 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:59.904544 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:59.904926 (+ 382us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:59.905199 (+ 273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:58:59.905205 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:58:59.905210 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:58:59.905400 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:58:59.905764 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:58:59.905773 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:58:59.906115 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 18:58:59.906387 (+ 272us) client_negotiation.cc:240] Negotiation successful
0810 18:58:59.906703 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":113,"threads_started":1}
I20260810 18:58:59.908694 5507 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42793
I20260810 18:58:59.908942 5507 heartbeater.cc:464] Registering TS with master...
I20260810 18:58:59.909417 5507 heartbeater.cc:511] Master 127.27.147.190:42793 requested a full tablet report, sending...
I20260810 18:58:59.910507 5024 ts_manager.cc:194] Registered new tserver with Master: c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130:44075)
I20260810 18:58:59.911396 5024 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:41353
WARNING: no policy specified for oryx/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab.
Entry for principal oryx/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab.
I20260810 18:58:59.931424 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42793
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab
--principal=oryx/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:00.057379 5513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:00.057615 5513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:00.057657 5513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:00.061661 5513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:00.061713 5513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:00.061769 5513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:00.066654 5513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab
--principal=oryx/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42793
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:00.067628 5513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:00.068552 5513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:00.078169 5518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.078222 5513 server_base.cc:1034] running on GCE node
W20260810 18:59:00.078158 5519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.078145 5521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.078766 5513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:00.079403 5513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:00.080562 5513 hybrid_clock.cc:648] HybridClock initialized: now 1786388340080548 us; error 23 us; skew 500 ppm
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:00.083308 5513 init.cc:377] Logged in from keytab as oryx/127.27.147.131@KRBTEST.COM (short username oryx)
I20260810 18:59:00.084134 5513 webserver.cc:460] Webserver started at http://127.27.147.131:39731/ using document root <none> and password file <none>
I20260810 18:59:00.084690 5513 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.084751 5513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.084995 5513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:00.087054 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "2a533e704c5e4f4eaf7851abf9828d86"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "f93ed64f7cc3322142ff44f0bb5758e5"
server_key_iv: "e8a6cc6178b2dcbdd3c0d6546547b312"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.087617 5513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "2a533e704c5e4f4eaf7851abf9828d86"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "f93ed64f7cc3322142ff44f0bb5758e5"
server_key_iv: "e8a6cc6178b2dcbdd3c0d6546547b312"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.091320 5513 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:59:00.093636 5532 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.094779 5513 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:00.094933 5513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "2a533e704c5e4f4eaf7851abf9828d86"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "f93ed64f7cc3322142ff44f0bb5758e5"
server_key_iv: "e8a6cc6178b2dcbdd3c0d6546547b312"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.095024 5513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.114370 5513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:00.117152 5513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:00.117302 5513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:00.117835 5513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:00.118624 5513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:00.118680 5513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.118718 5513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:00.118739 5513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.127549 5513 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:38435
I20260810 18:59:00.127575 5692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:38435 every 8 connection(s)
I20260810 18:59:00.128444 5513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:00.131726 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 5513
I20260810 18:59:00.131822 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:59:00.132105 28238 external_mini_cluster.cc:1428] Setting key d314fc6556e9180b68d56eda917d72cf
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:00.141530 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.130072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:42921 (local address 127.27.147.190:42793)
0810 18:59:00.130212 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.130217 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.131039 (+ 822us) server_negotiation.cc:407] Connection header received
0810 18:59:00.132506 (+ 1467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.132509 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.132574 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.132683 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.133932 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.134465 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.135632 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.135859 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.135976 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.138271 (+ 2295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.138274 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.138277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.138299 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.139855 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.140310 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.140313 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.140314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.140385 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.140663 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.140665 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.140667 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.140825 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.140898 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.141314 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.141405 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:59:00.142279 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.130423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:42921)
0810 18:59:00.130900 (+ 477us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.130942 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.132301 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.132855 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.132861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.133311 (+ 450us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.133794 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.133807 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.134592 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.134596 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.135509 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.135519 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.136007 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.136011 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.136175 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.136664 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.136680 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.138113 (+ 1433us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.139950 (+ 1837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.139955 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.139970 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.140226 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.140459 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.140461 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.140463 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.140588 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.140906 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.140911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.141125 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.141271 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.141501 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":109,"threads_started":1}
I20260810 18:59:00.143368 5694 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42793
I20260810 18:59:00.143594 5694 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:00.144052 5694 heartbeater.cc:511] Master 127.27.147.190:42793 requested a full tablet report, sending...
I20260810 18:59:00.145162 5025 ts_manager.cc:194] Registered new tserver with Master: 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131:38435)
I20260810 18:59:00.145910 5025 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:42921
I20260810 18:59:00.145920 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.147.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.189.keytab.
Entry for principal oryx/127.27.147.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.189.keytab.
I20260810 18:59:00.173836 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:43471
--webserver_interface=127.27.147.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.189.keytab
--principal=oryx/127.27.147.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:42793,127.27.147.189:43471
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:00.286141 5703 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:00.286387 5703 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:00.286465 5703 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:00.290329 5703 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:00.290382 5703 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:00.290402 5703 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:00.290447 5703 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:00.290479 5703 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:00.295478 5703 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:42793,127.27.147.189:43471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.189.keytab
--principal=oryx/127.27.147.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:43471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.147.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5703
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
W20260810 18:59:00.295672 5703 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.27.147.190:42793,127.27.147.189:43471'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 18:59:00.296469 5703 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:00.297427 5703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:00.305903 5711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.305999 5703 server_base.cc:1034] running on GCE node
W20260810 18:59:00.305904 5713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.305930 5710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.306777 5703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:00.307839 5703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:00.309034 5703 hybrid_clock.cc:648] HybridClock initialized: now 1786388340309018 us; error 35 us; skew 500 ppm
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:00.311887 5703 init.cc:377] Logged in from keytab as oryx/127.27.147.189@KRBTEST.COM (short username oryx)
I20260810 18:59:00.312892 5703 webserver.cc:460] Webserver started at http://127.27.147.189:42909/ using document root <none> and password file <none>
I20260810 18:59:00.313534 5703 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.313625 5703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.313831 5703 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:00.315883 5703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/instance:
uuid: "f697e7c944a94e18bbfdb8c370670a63"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "fbe05badbce84e07676ee3c20ad09d1e"
server_key_iv: "e40b3f6354c692c0929d910e67d52f08"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.316465 5703 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal/instance:
uuid: "f697e7c944a94e18bbfdb8c370670a63"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "fbe05badbce84e07676ee3c20ad09d1e"
server_key_iv: "e40b3f6354c692c0929d910e67d52f08"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.320173 5703 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:59:00.322551 5726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.323835 5703 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:00.323987 5703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
uuid: "f697e7c944a94e18bbfdb8c370670a63"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "fbe05badbce84e07676ee3c20ad09d1e"
server_key_iv: "e40b3f6354c692c0929d910e67d52f08"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.324095 5703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.339612 5703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:00.342233 5703 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:00.342427 5703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:00.349884 5703 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.189:43471
I20260810 18:59:00.350140 5794 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.189:43471 every 8 connection(s)
I20260810 18:59:00.350939 5703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
I20260810 18:59:00.353582 5795 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:00.357254 5795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
I20260810 18:59:00.360368 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 5703
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.189@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.147.189@KRBTEST.COM
I20260810 18:59:00.371201 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.359040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34500 (local address 127.27.147.190:42793)
0810 18:59:00.359200 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.359204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.359997 (+ 793us) server_negotiation.cc:407] Connection header received
0810 18:59:00.360964 (+ 967us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.360968 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.361027 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.361121 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.362550 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.363341 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.364652 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.364860 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.364991 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.367395 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.367404 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.367407 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.367442 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.369024 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.369535 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.369539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.369541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.369623 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.370030 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.370033 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.370035 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.370247 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.370354 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.370894 (+ 540us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.371027 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 18:59:00.371906 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.359383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34500)
0810 18:59:00.359850 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.359900 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.360761 (+ 861us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.361296 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.361305 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.361778 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.362375 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.362395 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.363476 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.363479 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.364533 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.364545 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.365026 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.365029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.365175 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.365751 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.365777 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.367253 (+ 1476us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.369142 (+ 1889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.369151 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.369160 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.369426 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.369715 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.369719 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.369721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.369924 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.370353 (+ 429us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.370359 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.370632 (+ 273us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.370802 (+ 170us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.371120 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":78,"thread_start_us":114,"threads_started":1}
I20260810 18:59:00.371906 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.361190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.27.147.189:43471)
0810 18:59:00.361534 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.361547 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.361567 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:59:00.361713 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.361718 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.361902 (+ 184us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.362025 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.362904 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.363940 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.364919 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.365359 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.365479 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.367861 (+ 2382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.367867 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.367876 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.367897 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.369369 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.369810 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.369816 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.369818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.369911 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.370233 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.370235 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.370236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.370610 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.370806 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.371074 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.371220 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":266,"spinlock_wait_cycles":25984,"thread_start_us":83,"threads_started":1}
I20260810 18:59:00.373466 5795 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.189@KRBTEST.COM for oryx/127.27.147.189@KRBTEST.COM
I20260810 18:59:00.381219 5799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.374172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.0.0.1:56698)
0810 18:59:00.374315 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.374328 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.374403 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.374595 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.374598 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.374745 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.374861 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.374868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.375559 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.375562 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.376049 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.376055 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.376483 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.376487 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.376574 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.376989 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.377002 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.378094 (+ 1092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.379820 (+ 1726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.379826 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.379828 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.380011 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.380287 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.380290 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.380292 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.380391 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.380733 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.380736 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.380816 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.380919 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.381051 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 18:59:00.381225 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.374253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56698 (local address 127.27.147.189:43471)
0810 18:59:00.374353 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.374359 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.374373 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:00.374474 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.374476 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.374525 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.374586 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.374962 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.375440 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.376157 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.376376 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.376479 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.378213 (+ 1734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.378217 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.378219 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.378245 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.379718 (+ 1473us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.380104 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.380109 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.380111 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.380192 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.380483 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.380487 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.380488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.380643 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.380742 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.380982 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.381104 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":39}
I20260810 18:59:00.384413 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Bootstrap starting.
I20260810 18:59:00.386588 5795 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:00.387338 5795 log.cc:824] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:00.389056 5795 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: No bootstrap required, opened a new log
I20260810 18:59:00.391993 5795 raft_consensus.cc:348] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } }
I20260810 18:59:00.392197 5795 raft_consensus.cc:374] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:00.392284 5795 raft_consensus.cc:729] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f697e7c944a94e18bbfdb8c370670a63, State: Initialized, Role: FOLLOWER
I20260810 18:59:00.392691 5795 consensus_queue.cc:260] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } }
I20260810 18:59:00.393142 5804 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } } }
I20260810 18:59:00.393267 5804 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:59:00.393810 5795 sys_catalog.cc:564] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:59:00.398502 5818 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:59:00.398589 5818 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:59:00.402432 5703 master_runner.cc:428] Detected that this master f697e7c944a94e18bbfdb8c370670a63 is joining an existing cluster
I20260810 18:59:00.402516 5703 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.147.189:43471
I20260810 18:59:00.414425 5832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.405057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34504)
0810 18:59:00.405824 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.405840 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.405936 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.406748 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.406751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.407002 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.407135 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.407143 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.408181 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.408185 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.408870 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.408881 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.409434 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.409438 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.409540 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.410126 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.410142 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.410516 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.412674 (+ 2158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.412679 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.412681 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.413005 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.413317 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.413320 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.413322 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.413411 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.413874 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.413877 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.413981 (+ 104us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.414086 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.414220 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":131,"threads_started":1}
I20260810 18:59:00.414503 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.405496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34504 (local address 127.27.147.190:42793)
0810 18:59:00.406445 (+ 949us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.406450 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.406463 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:00.406499 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.406502 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.406547 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.406623 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.407272 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.408069 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.408984 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.409326 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.409437 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.410631 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.410640 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.410642 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.410665 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.412490 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.413119 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.413123 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.413125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.413217 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.413535 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.413538 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.413540 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.413776 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.413884 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.414157 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.414334 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865}
I20260810 18:59:00.414592 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.405355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56702 (local address 127.27.147.189:43471)
0810 18:59:00.406094 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.406098 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.406109 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:59:00.406138 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.406143 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.406186 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.406270 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.406879 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.407666 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.408891 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.409157 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.409264 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.410590 (+ 1326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.410595 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.410598 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.410624 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.412648 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.413193 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.413196 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.413198 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.413308 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.413573 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.413579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.413582 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.413856 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.413938 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.414330 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.414430 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":672}
I20260810 18:59:00.415022 5828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.404939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.0.0.1:56702)
0810 18:59:00.405576 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.405594 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.405704 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.406316 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.406321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.406570 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.406719 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.406726 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.407791 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.407794 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.408763 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.408777 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.409275 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.409278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.409408 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.409925 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.409937 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.410433 (+ 496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.412753 (+ 2320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.412759 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.412763 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.413090 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.413394 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.413397 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.413399 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.413495 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.414007 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.414010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.414101 (+ 91us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.414738 (+ 637us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.414905 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":91,"threads_started":1}
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.190@KRBTEST.COM for oryx/127.27.147.189@KRBTEST.COM
I20260810 18:59:00.428515 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.419131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.0.0.1:56716)
0810 18:59:00.419476 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.419533 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.419661 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.419940 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.419944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.420251 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.420346 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.420351 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.421272 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.421274 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.421881 (+ 607us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.421889 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.422432 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.422434 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.422519 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.423043 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.423056 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.424589 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.426593 (+ 2004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.426603 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.426609 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.426960 (+ 351us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.427330 (+ 370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.427333 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.427335 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.427431 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.427902 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.427909 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.428023 (+ 114us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.428165 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.428345 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":105,"threads_started":1}
I20260810 18:59:00.428550 5801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.419162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56716 (local address 127.27.147.189:43471)
0810 18:59:00.419268 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.419272 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.419610 (+ 338us) server_negotiation.cc:407] Connection header received
0810 18:59:00.419801 (+ 191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.419804 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.419851 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.419921 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.420448 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.421156 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.421998 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.422325 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.422428 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.424766 (+ 2338us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.424769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.424771 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.424791 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.426472 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.427069 (+ 597us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.427076 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.427080 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.427197 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.427519 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.427525 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.427530 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.427767 (+ 237us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.427882 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.428219 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.428341 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:59:00.430102 5025 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.147.189:43471: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63"
member_type: NON_VOTER
last_known_addr {
host: "127.27.147.189"
port: 43471
}
attrs {
promote: true
}
}
dest_uuid: "b007540de45341a0a6c5ef083c571502"
cas_config_opid_index: -1
I20260810 18:59:00.430712 5025 consensus_queue.cc:237] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } }
I20260810 18:59:00.432708 5838 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } }
I20260810 18:59:00.433344 5762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:00.434217 5838 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.147.189:43471
I20260810 18:59:00.434439 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } }
I20260810 18:59:00.434558 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:00.434696 5703 master.cc:537] Master@127.27.147.189:43471 shutting down...
I20260810 18:59:00.434710 5762 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Refusing update from remote peer b007540de45341a0a6c5ef083c571502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 18:59:00.437860 5838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:00.439021 5703 raft_consensus.cc:2229] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:00.439296 5703 raft_consensus.cc:2258] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:00.439421 5703 tablet_replica.cc:330] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: stopping tablet replica
W20260810 18:59:00.440034 5004 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 18:59:00.440901 5004 consensus_peers.cc:487] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 -> Peer f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189:43471): Couldn't send request to peer f697e7c944a94e18bbfdb8c370670a63. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for oryx/127.27.147.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.188@KRBTEST.COM" created.
I20260810 18:59:00.445813 5703 master.cc:552] Master@127.27.147.189:43471 shutdown complete.
I20260810 18:59:00.445871 5703 master_runner.cc:305] Clearing existing system tablet
I20260810 18:59:00.446098 5703 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.446168 5703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.447293 5703 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260810 18:59:00.448315 5846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.448518 5703 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.448868 5703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 18:59:00.448963 5703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
uuid: "f697e7c944a94e18bbfdb8c370670a63"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "fbe05badbce84e07676ee3c20ad09d1e"
server_key_iv: "e40b3f6354c692c0929d910e67d52f08"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.450562 5703 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:00.452741 5703 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:59:00.452847 5703 log.cc:1196] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 18:59:00.453086 5703 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Deleting consensus metadata
I20260810 18:59:00.453305 5703 master_runner.cc:315] Copying system tablet from 127.27.147.190:42793
I20260810 18:59:00.454732 5703 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: tablet copy: Beginning tablet copy session from remote peer at address 127.27.147.190:42793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.188.keytab.
Entry for principal oryx/127.27.147.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.188.keytab.
I20260810 18:59:00.466070 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:32963
--webserver_interface=127.27.147.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.188.keytab
--principal=oryx/127.27.147.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:59:00.466400 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.455414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34512 (local address 127.27.147.190:42793)
0810 18:59:00.455547 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.455553 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.458106 (+ 2553us) server_negotiation.cc:407] Connection header received
0810 18:59:00.458220 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.458223 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.458272 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.458346 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.459017 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.459727 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.461022 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.461238 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.461317 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.463169 (+ 1852us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.463173 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.463177 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.463208 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.464631 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.465102 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.465156 (+ 54us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.465158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.465254 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.465582 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.465586 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.465587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.465762 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.465866 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.466134 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.466244 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:59:00.466413 5861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.455318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34512)
0810 18:59:00.458000 (+ 2682us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.458015 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.458134 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.458383 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.458386 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.458637 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.458874 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.458884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.459846 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.459849 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.460866 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.460878 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.461363 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.461367 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.461466 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.462577 (+ 1111us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.462589 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.463034 (+ 445us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.464751 (+ 1717us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.464754 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.464756 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.464980 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.465371 (+ 391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.465377 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.465382 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.465479 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.465857 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.465860 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.465963 (+ 103us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.466064 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.466206 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2600,"thread_start_us":81,"threads_started":1}
I20260810 18:59:00.466975 5058 tablet_copy_service.cc:140] P b007540de45341a0a6c5ef083c571502: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer f697e7c944a94e18bbfdb8c370670a63 ({username='oryx', principal='oryx/127.27.147.189@KRBTEST.COM'} at 127.0.0.1:34512)
I20260810 18:59:00.467175 5058 tablet_copy_service.cc:161] P b007540de45341a0a6c5ef083c571502: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer f697e7c944a94e18bbfdb8c370670a63 at {username='oryx', principal='oryx/127.27.147.189@KRBTEST.COM'} at 127.0.0.1:34512: session id = f697e7c944a94e18bbfdb8c370670a63-00000000000000000000000000000000
I20260810 18:59:00.468587 5058 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:59:00.470644 5703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:59:00.473757 5703 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: tablet copy: Starting download of 0 data blocks...
I20260810 18:59:00.473981 5703 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: tablet copy: Starting download of 1 WAL segments...
I20260810 18:59:00.475891 5703 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:59:00.478045 5058 tablet_copy_service.cc:342] P b007540de45341a0a6c5ef083c571502: Request end of tablet copy session f697e7c944a94e18bbfdb8c370670a63-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.147.189@KRBTEST.COM'} at 127.0.0.1:34512
I20260810 18:59:00.478168 5058 tablet_copy_service.cc:433] P b007540de45341a0a6c5ef083c571502: ending tablet copy session f697e7c944a94e18bbfdb8c370670a63-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer f697e7c944a94e18bbfdb8c370670a63
I20260810 18:59:00.486063 5703 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:00.487730 5863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.487744 5868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.488369 5703 server_base.cc:1034] running on GCE node
W20260810 18:59:00.490983 5865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.491196 5703 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:00.491488 5703 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:00.492635 5703 hybrid_clock.cc:648] HybridClock initialized: now 1786388340492613 us; error 38 us; skew 500 ppm
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:00.494261 5703 init.cc:377] Logged in from keytab as oryx/127.27.147.189@KRBTEST.COM (short username oryx)
I20260810 18:59:00.494714 5703 webserver.cc:460] Webserver started at http://127.27.147.189:33865/ using document root <none> and password file <none>
I20260810 18:59:00.494927 5703 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.494982 5703 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.496260 5703 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260810 18:59:00.497285 5875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.497527 5703 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260810 18:59:00.497641 5703 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
uuid: "f697e7c944a94e18bbfdb8c370670a63"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "fbe05badbce84e07676ee3c20ad09d1e"
server_key_iv: "e40b3f6354c692c0929d910e67d52f08"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.497779 5703 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.525870 5703 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:00.528280 5703 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:00.533743 5703 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.189:43471
I20260810 18:59:00.533794 5963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.189:43471 every 8 connection(s)
I20260810 18:59:00.534512 5703 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
I20260810 18:59:00.536433 5966 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:59:00.537443 5966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Bootstrap starting.
I20260810 18:59:00.548065 5966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:59:00.548395 5966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: Bootstrap complete.
I20260810 18:59:00.549019 5966 raft_consensus.cc:348] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } }
I20260810 18:59:00.549162 5966 raft_consensus.cc:729] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f697e7c944a94e18bbfdb8c370670a63, State: Initialized, Role: LEARNER
I20260810 18:59:00.549261 5966 consensus_queue.cc:260] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } }
I20260810 18:59:00.549604 5969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } }
I20260810 18:59:00.549705 5966 sys_catalog.cc:564] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:00.549814 5969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:59:00.554133 5985 catalog_manager.cc:1259] Loaded cluster ID: e659b0ff0a5c428caa87ff428685bbf3
I20260810 18:59:00.554191 5985 catalog_manager.cc:1552] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: loading cluster ID for follower catalog manager: success
I20260810 18:59:00.556383 5985 catalog_manager.cc:1574] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: acquiring CA information for follower catalog manager: success
I20260810 18:59:00.557674 5985 catalog_manager.cc:1602] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 18:59:00.597736 5862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:00.598044 5862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:00.598117 5862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:00.602226 5862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:00.602277 5862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:00.602298 5862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:00.602313 5862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:00.602329 5862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:00.607300 5862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.188.keytab
--principal=oryx/127.27.147.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:32963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.147.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:00.608201 5862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:00.609130 5862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:00.617532 5992 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.617591 5862 server_base.cc:1034] running on GCE node
W20260810 18:59:00.617558 5994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.617725 5998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.618289 5862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:00.619379 5862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:00.620532 5862 hybrid_clock.cc:648] HybridClock initialized: now 1786388340620502 us; error 42 us; skew 500 ppm
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:00.623194 5862 init.cc:377] Logged in from keytab as oryx/127.27.147.188@KRBTEST.COM (short username oryx)
I20260810 18:59:00.624119 5862 webserver.cc:460] Webserver started at http://127.27.147.188:45251/ using document root <none> and password file <none>
I20260810 18:59:00.624648 5862 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.624737 5862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.624998 5862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:00.626962 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/instance:
uuid: "8c2b5bc1db18407da8e34c713db85f36"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "82a7f9ac792cb532878dcb18ad70cfcb"
server_key_iv: "7148e15e9b3648bf4be9b7bdf81b66f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.627501 5862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal/instance:
uuid: "8c2b5bc1db18407da8e34c713db85f36"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "82a7f9ac792cb532878dcb18ad70cfcb"
server_key_iv: "7148e15e9b3648bf4be9b7bdf81b66f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.631039 5862 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:59:00.633319 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.634470 5862 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:00.634601 5862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
uuid: "8c2b5bc1db18407da8e34c713db85f36"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "82a7f9ac792cb532878dcb18ad70cfcb"
server_key_iv: "7148e15e9b3648bf4be9b7bdf81b66f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.634703 5862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.649340 5862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:00.652918 5862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:00.653139 5862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:00.662370 5862 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.188:32963
I20260810 18:59:00.662708 6081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.188:32963 every 8 connection(s)
I20260810 18:59:00.663607 5862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
I20260810 18:59:00.665585 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 5862
I20260810 18:59:00.666983 6082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:00.672112 6082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.147.188@KRBTEST.COM
I20260810 18:59:00.680167 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.667183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.27.147.188:32963)
0810 18:59:00.667608 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.667623 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.667658 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:59:00.668272 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.668287 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.668597 (+ 310us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.668915 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.669708 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.670861 (+ 1153us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.671745 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.672220 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.672432 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.675155 (+ 2723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.675161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.675175 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.675206 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.677297 (+ 2091us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.677691 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.677697 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.677703 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.677819 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.678106 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.678111 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.678113 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.678598 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.678879 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.679177 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.679511 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":294,"thread_start_us":147,"threads_started":1}
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.188@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:00.683925 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.673948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34522 (local address 127.27.147.190:42793)
0810 18:59:00.674082 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.674089 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.674913 (+ 824us) server_negotiation.cc:407] Connection header received
0810 18:59:00.675145 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.675149 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.675207 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.675300 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.676366 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.677192 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.678197 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.678514 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.678630 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.680814 (+ 2184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.680818 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.680821 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.680848 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.682436 (+ 1588us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.682860 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.682864 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.682865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.682945 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.683184 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.683186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.683188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.683356 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.683432 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.683726 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.683817 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:59:00.683933 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.673829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34522)
0810 18:59:00.674756 (+ 927us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.674806 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.675007 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.675345 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.675351 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.675832 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.676002 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.676010 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.677323 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.677326 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.678080 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.678090 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.678624 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.678626 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.678759 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.679298 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.679313 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.680681 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.682537 (+ 1856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.682541 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.682545 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.682749 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.683031 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.683033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.683034 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.683108 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.683441 (+ 333us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.683446 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.683557 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.683668 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.683789 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"thread_start_us":138,"threads_started":1}
I20260810 18:59:00.684886 6082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.188@KRBTEST.COM for oryx/127.27.147.189@KRBTEST.COM
I20260810 18:59:00.692000 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.685331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.0.0.1:56728)
0810 18:59:00.685444 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.685458 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.685544 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.685888 (+ 344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.685891 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.686013 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.686095 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.686100 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.686984 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.686986 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.687483 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.687490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.687872 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.687875 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.687945 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.688298 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.688309 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.689387 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.690874 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.690877 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.690878 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.691055 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.691288 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.691290 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.691291 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.691359 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.691665 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.691667 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.691736 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.691812 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.691900 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 18:59:00.692122 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.685453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56728 (local address 127.27.147.189:43471)
0810 18:59:00.685700 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.685704 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.685715 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:59:00.685751 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.685754 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.685791 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.685862 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.686192 (+ 330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.686894 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.687577 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.687788 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.687859 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.689519 (+ 1660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.689522 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.689524 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.689545 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.690746 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.691137 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.691140 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.691141 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.691212 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.691431 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.691433 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.691434 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.691592 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.691664 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.691869 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.691962 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":179,"thread_start_us":97,"threads_started":1}
I20260810 18:59:00.692778 6082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.188@KRBTEST.COM for oryx/127.27.147.188@KRBTEST.COM
I20260810 18:59:00.697968 5320 heartbeater.cc:503] Master 127.27.147.190:42793 was elected leader, sending a full tablet report...
I20260810 18:59:00.700115 6086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.693190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.0.0.1:58728)
0810 18:59:00.693291 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.693305 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.693369 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.693524 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.693526 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.693640 (+ 114us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.693703 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.693707 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.694690 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.694692 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.695212 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.695220 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.695609 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.695611 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.695667 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.695998 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.696009 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.697027 (+ 1018us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.698800 (+ 1773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.698802 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.698803 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.699036 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.699392 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.699394 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.699396 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.699478 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.699794 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.699796 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.699863 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.699938 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.700022 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260810 18:59:00.700218 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.693253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58728 (local address 127.27.147.188:32963)
0810 18:59:00.693352 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.693355 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.693373 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:59:00.693423 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.693425 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.693456 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.693511 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.693785 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.694586 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.695315 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.695524 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.695594 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.697150 (+ 1556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.697153 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.697155 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.697174 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.698705 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.699190 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.699193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.699194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.699294 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.699561 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.699563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.699565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.699715 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.699789 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.699999 (+ 210us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.700079 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:59:00.702756 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Bootstrap starting.
I20260810 18:59:00.704916 6082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:00.705576 6082 log.cc:824] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:00.707258 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: No bootstrap required, opened a new log
I20260810 18:59:00.710063 6082 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } }
I20260810 18:59:00.710233 6082 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:00.710280 6082 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c2b5bc1db18407da8e34c713db85f36, State: Initialized, Role: FOLLOWER
I20260810 18:59:00.710626 6082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } }
I20260810 18:59:00.711063 6090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } } }
I20260810 18:59:00.711184 6090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:59:00.711759 6082 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 18:59:00.716361 6110 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 18:59:00.716435 6110 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 18:59:00.719225 5862 master_runner.cc:428] Detected that this master 8c2b5bc1db18407da8e34c713db85f36 is joining an existing cluster
I20260810 18:59:00.719287 5862 master_runner.cc:432] Initiating AddMaster RPC to add 127.27.147.188:32963
I20260810 18:59:00.733557 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 5120
I20260810 18:59:00.749204 6118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.721197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.0.0.1:56744)
0810 18:59:00.721557 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.721570 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.721654 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.722060 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.722063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.722386 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.722535 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.722544 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.723622 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.723626 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.724675 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.724685 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.725364 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.725368 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.725478 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.725926 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.725937 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.726262 (+ 325us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.728121 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.728126 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.728129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.747672 (+ 19543us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.748011 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.748016 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.748021 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.748139 (+ 118us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.748584 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.748589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.748691 (+ 102us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.748779 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.748944 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"mutex_wait_us":1,"thread_start_us":111,"threads_started":1}
I20260810 18:59:00.749930 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.721703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56744 (local address 127.27.147.189:43471)
0810 18:59:00.721840 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.721845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.721858 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:00.721894 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.721896 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.721937 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.722020 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.722669 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.723496 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.724811 (+ 1315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.725144 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.725233 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.726395 (+ 1162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.726399 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.726402 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.726425 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.727992 (+ 1567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.747804 (+ 19812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.747811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.747815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.747905 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.748231 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.748236 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.748240 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.748460 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.748543 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.749598 (+ 1055us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.749754 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 18:59:00.753983 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.722316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58738 (local address 127.27.147.188:32963)
0810 18:59:00.722431 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.722435 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.722881 (+ 446us) server_negotiation.cc:407] Connection header received
0810 18:59:00.723043 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.723046 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.723083 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.723127 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.723679 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.724384 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.725563 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.725891 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.725983 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.727291 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.727295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.727298 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.727323 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.728850 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.750081 (+ 21231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.750086 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.750088 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.750185 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.750783 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.750786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.750787 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.751016 (+ 229us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.751129 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.753675 (+ 2546us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.753779 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:59:00.754436 6119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.722295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.0.0.1:58738)
0810 18:59:00.722788 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.722801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.722944 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.723241 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.723255 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.723435 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.723560 (+ 125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.723568 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.724507 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.724510 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.725441 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.725452 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.726113 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.726117 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.726205 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.726642 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.726661 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.727132 (+ 471us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.728945 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.728947 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.728948 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.749399 (+ 20451us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.750315 (+ 916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.750318 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.750320 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.750413 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.751206 (+ 793us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.751211 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.751304 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.754085 (+ 2781us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.754215 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":90,"threads_started":1}
I20260810 18:59:00.756050 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.721595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34538 (local address 127.27.147.190:42793)
0810 18:59:00.721840 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.721845 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.721858 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:00.721894 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.721897 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.721937 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.722020 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.723059 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.723793 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.724954 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.725291 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.725365 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.751485 (+ 26120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.751490 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.751492 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.751524 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.753168 (+ 1644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.754668 (+ 1500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.754674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.754678 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.754763 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.755069 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.755074 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.755078 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.755294 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.755367 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.755717 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.755842 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178}
I20260810 18:59:00.756567 6116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.721166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34538)
0810 18:59:00.721511 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.721526 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.721616 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.722536 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.722538 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.722758 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.722909 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.722917 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.723925 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.723928 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.724842 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.724850 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.725505 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.725508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.725620 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.749703 (+ 24083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.749721 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.750645 (+ 924us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.753303 (+ 2658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.753308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.753313 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.753542 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.754891 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.754897 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.754901 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.754981 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.755437 (+ 456us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.755442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.755548 (+ 106us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.756175 (+ 627us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.756323 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":117,"threads_started":1}
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388339, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.190@KRBTEST.COM for oryx/127.27.147.188@KRBTEST.COM
I20260810 18:59:00.772217 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:37149
--local_ip_for_outbound_sockets=127.27.147.129
--tserver_master_addrs=127.27.147.190:42793
--webserver_port=38713
--webserver_interface=127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab
--principal=oryx/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:59:00.774251 6084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.764540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58748 (local address 127.27.147.188:32963)
0810 18:59:00.764689 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.764696 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.764713 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:59:00.764783 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.764789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.764851 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.764935 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.765578 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.766408 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.767568 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.767979 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.768098 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.770574 (+ 2476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.770581 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.770587 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.770617 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.772397 (+ 1780us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.772937 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.772941 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.772943 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.773041 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.773357 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.773361 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.773363 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.773564 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.773661 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.773989 (+ 328us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.774101 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:59:00.774304 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.764083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.0.0.1:58748)
0810 18:59:00.764325 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.764338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.764413 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.765088 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.765094 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.765305 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.765436 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.765446 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.766538 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.766542 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.767436 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.767448 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.768104 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.768108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.768211 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.768712 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.768727 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.770394 (+ 1667us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.772536 (+ 2142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.772542 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.772546 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.772815 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.773147 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.773153 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.773158 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.773260 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.773669 (+ 409us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.773675 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.773793 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.773917 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.774084 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166}
I20260810 18:59:00.775558 5025 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.27.147.188:32963: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36"
member_type: NON_VOTER
last_known_addr {
host: "127.27.147.188"
port: 32963
}
attrs {
promote: true
}
}
dest_uuid: "b007540de45341a0a6c5ef083c571502"
cas_config_opid_index: 5
I20260810 18:59:00.776273 5025 consensus_queue.cc:237] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } }
I20260810 18:59:00.777580 5836 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.779476 5836 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.27.147.188:32963
I20260810 18:59:00.779680 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.779814 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:00.780601 5862 master.cc:537] Master@127.27.147.188:32963 shutting down...
I20260810 18:59:00.780947 6043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:00.782380 6043 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Refusing update from remote peer b007540de45341a0a6c5ef083c571502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 18:59:00.783715 5838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:59:00.785727 5006 consensus_peers.cc:487] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 -> Peer 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188:32963): Couldn't send request to peer 8c2b5bc1db18407da8e34c713db85f36. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 18:59:00.788676 5862 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:00.789021 5862 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:00.789193 5862 tablet_replica.cc:330] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: stopping tablet replica
I20260810 18:59:00.792940 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.777714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.0.0.1:56754)
0810 18:59:00.787647 (+ 9933us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.787662 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.787747 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.788396 (+ 649us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.788399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.788416 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:00.788508 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.788621 (+ 113us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.789671 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.789674 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.790971 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.790982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.792260 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.792263 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.792621 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.792672 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.792748 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.792816 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9854}
I20260810 18:59:00.793321 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.777945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.27.147.189:43471)
0810 18:59:00.778142 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.778146 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.788063 (+ 9917us) server_negotiation.cc:407] Connection header received
0810 18:59:00.788121 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.788125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.788185 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.788272 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:00.788739 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.789545 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.791287 (+ 1742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.791960 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.792075 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.792196 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.793131 (+ 935us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.793209 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260810 18:59:00.793792 5918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 LEARNER]: Refusing update from remote peer b007540de45341a0a6c5ef083c571502: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 18:59:00.794212 5838 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 18:59:00.796166 5918 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.798606 5969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b007540de45341a0a6c5ef083c571502. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.798751 5969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:59:00.798921 5969 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: NON_VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: true } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.799026 5969 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:59:00.799685 5838 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: attempting to promote NON_VOTER f697e7c944a94e18bbfdb8c370670a63 to VOTER
I20260810 18:59:00.800118 5838 consensus_queue.cc:237] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } }
I20260810 18:59:00.801541 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.801270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.0.0.1:58764)
0810 18:59:00.801399 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.801474 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 18:59:00.801920 5918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 LEARNER]: Refusing update from remote peer b007540de45341a0a6c5ef083c571502: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 18:59:00.802395 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.802146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.0.0.1:58766)
0810 18:59:00.802294 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.802337 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:32963: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 18:59:00.802628 5006 consensus_peers.cc:487] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 -> Peer 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188:32963): Couldn't send request to peer 8c2b5bc1db18407da8e34c713db85f36. Status: Network error: Client connection negotiation failed: client connection to 127.27.147.188:32963: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:59:00.802708 6126 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 18:59:00.804562 5862 master.cc:552] Master@127.27.147.188:32963 shutdown complete.
I20260810 18:59:00.804611 5862 master_runner.cc:305] Clearing existing system tablet
I20260810 18:59:00.804849 5862 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.804896 5862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.806200 5862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:00.806944 5836 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.808466 5836 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.808562 5836 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:00.809373 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.809485 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:00.809471 5918 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.809811 6131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.810027 5862 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.810138 5862 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 18:59:00.810170 5862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
uuid: "8c2b5bc1db18407da8e34c713db85f36"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "82a7f9ac792cb532878dcb18ad70cfcb"
server_key_iv: "7148e15e9b3648bf4be9b7bdf81b66f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.811010 6125 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.811113 6125 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:59:00.812196 5862 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:00.814968 5862 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 18:59:00.815114 5862 log.cc:1196] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 18:59:00.815438 5862 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Deleting consensus metadata
I20260810 18:59:00.815682 5862 master_runner.cc:315] Copying system tablet from 127.27.147.190:42793
I20260810 18:59:00.816975 5862 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: tablet copy: Beginning tablet copy session from remote peer at address 127.27.147.190:42793
I20260810 18:59:00.826174 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.817611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34540 (local address 127.27.147.190:42793)
0810 18:59:00.817725 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.817730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.817892 (+ 162us) server_negotiation.cc:407] Connection header received
0810 18:59:00.818020 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.818023 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.818072 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.818158 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.818756 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.819585 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.820794 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.821117 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.821229 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.822541 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.822547 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.822552 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.822582 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.824409 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.824890 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.824894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.824896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.824988 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:00.825275 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:00.825278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:00.825280 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:00.825503 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:00.825605 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:00.825901 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 18:59:00.826025 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:59:00.826215 6146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.817528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34540)
0810 18:59:00.817797 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.817811 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.817920 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.818192 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.818195 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.818422 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.818557 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.818565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.819698 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.819702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.820677 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.820689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.821245 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.821249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.821356 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.821941 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.821956 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.822388 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.824523 (+ 2135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.824527 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.824529 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.824783 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.825082 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.825085 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.825087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.825179 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:00.825603 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:00.825607 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:00.825718 (+ 111us) client_negotiation.cc:769] Sending connection context
0810 18:59:00.825834 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 18:59:00.825980 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":72,"threads_started":1}
I20260810 18:59:00.826930 5058 tablet_copy_service.cc:140] P b007540de45341a0a6c5ef083c571502: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 8c2b5bc1db18407da8e34c713db85f36 ({username='oryx', principal='oryx/127.27.147.188@KRBTEST.COM'} at 127.0.0.1:34540)
I20260810 18:59:00.827041 5058 tablet_copy_service.cc:161] P b007540de45341a0a6c5ef083c571502: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 8c2b5bc1db18407da8e34c713db85f36 at {username='oryx', principal='oryx/127.27.147.188@KRBTEST.COM'} at 127.0.0.1:34540: session id = 8c2b5bc1db18407da8e34c713db85f36-00000000000000000000000000000000
I20260810 18:59:00.828099 5058 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 18:59:00.829321 5862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 18:59:00.831822 5862 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: tablet copy: Starting download of 0 data blocks...
I20260810 18:59:00.832091 5862 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: tablet copy: Starting download of 1 WAL segments...
I20260810 18:59:00.833813 5862 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 18:59:00.835639 5058 tablet_copy_service.cc:342] P b007540de45341a0a6c5ef083c571502: Request end of tablet copy session 8c2b5bc1db18407da8e34c713db85f36-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.27.147.188@KRBTEST.COM'} at 127.0.0.1:34540
I20260810 18:59:00.835723 5058 tablet_copy_service.cc:433] P b007540de45341a0a6c5ef083c571502: ending tablet copy session 8c2b5bc1db18407da8e34c713db85f36-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 8c2b5bc1db18407da8e34c713db85f36
I20260810 18:59:00.843103 5862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:00.845111 6147 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.845124 6148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.845577 6151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.845783 5862 server_base.cc:1034] running on GCE node
I20260810 18:59:00.845928 5862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:00.846289 5862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:00.847431 5862 hybrid_clock.cc:648] HybridClock initialized: now 1786388340847422 us; error 51 us; skew 500 ppm
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:00.849026 5862 init.cc:377] Logged in from keytab as oryx/127.27.147.188@KRBTEST.COM (short username oryx)
I20260810 18:59:00.849478 5862 webserver.cc:460] Webserver started at http://127.27.147.188:43033/ using document root <none> and password file <none>
I20260810 18:59:00.849668 5862 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.849707 5862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.851184 5862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:00.852205 6164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.852490 5862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:00.852577 5862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
uuid: "8c2b5bc1db18407da8e34c713db85f36"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "82a7f9ac792cb532878dcb18ad70cfcb"
server_key_iv: "7148e15e9b3648bf4be9b7bdf81b66f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.852660 5862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.868356 5862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:00.870676 5862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:00.875941 5862 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.188:32963
I20260810 18:59:00.876093 6238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.188:32963 every 8 connection(s)
I20260810 18:59:00.876595 5862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
I20260810 18:59:00.878721 6239 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:59:00.880064 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Bootstrap starting.
I20260810 18:59:00.891712 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:59:00.891994 6239 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Bootstrap complete.
I20260810 18:59:00.892536 6239 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } }
I20260810 18:59:00.892668 6239 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8c2b5bc1db18407da8e34c713db85f36, State: Initialized, Role: LEARNER
I20260810 18:59:00.892763 6239 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } }
I20260810 18:59:00.893194 6239 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:00.896690 6264 catalog_manager.cc:1259] Loaded cluster ID: e659b0ff0a5c428caa87ff428685bbf3
I20260810 18:59:00.896749 6264 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: loading cluster ID for follower catalog manager: success
I20260810 18:59:00.898054 6242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:00.898278 6242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:59:00.899281 6264 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: acquiring CA information for follower catalog manager: success
I20260810 18:59:00.900693 6264 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 18:59:00.913506 5507 heartbeater.cc:503] Master 127.27.147.190:42793 was elected leader, sending a full tablet report...
W20260810 18:59:00.919440 6121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:00.919687 6121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:00.919766 6121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:00.924014 6121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:00.924091 6121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:00.924178 6121 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:59:00.929457 6121 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab
--principal=oryx/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:37149
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=38713
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6121
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:00.930449 6121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:00.931449 6121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:00.941200 6271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.941250 6270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:00.941251 6274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:00.941402 6121 server_base.cc:1034] running on GCE node
I20260810 18:59:00.941831 6121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:00.942413 6121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:00.943579 6121 hybrid_clock.cc:648] HybridClock initialized: now 1786388340943561 us; error 28 us; skew 500 ppm
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:00.946337 6121 init.cc:377] Logged in from keytab as oryx/127.27.147.129@KRBTEST.COM (short username oryx)
I20260810 18:59:00.947218 6121 webserver.cc:460] Webserver started at http://127.27.147.129:38713/ using document root <none> and password file <none>
I20260810 18:59:00.947769 6121 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:00.947878 6121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:00.951710 6121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:59:00.953776 6286 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.955024 6121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:00.955147 6121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "f34121eec10941dfbb6f650604c8656f"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "b5dfa05c06fd46b017892b51ab98c3ad"
server_key_iv: "943906e93ce5bc25272053bccd0f589d"
server_key_version: "encryptionkey@0"
I20260810 18:59:00.955658 6121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:00.968911 6121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:00.971594 6121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:00.971778 6121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:00.972321 6121 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:00.973152 6121 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:00.973219 6121 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.973285 6121 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:00.973331 6121 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:00.983533 6121 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:37149
I20260810 18:59:00.983575 6437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:37149 every 8 connection(s)
I20260810 18:59:00.984458 6121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:59:00.992293 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 6121
I20260810 18:59:00.992488 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 5328
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.189@KRBTEST.COM
Aug 10 18:59:00 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.188@KRBTEST.COM
I20260810 18:59:01.004194 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.986558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:44687 (local address 127.27.147.189:43471)
0810 18:59:00.986685 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.986690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.988248 (+ 1558us) server_negotiation.cc:407] Connection header received
0810 18:59:00.988577 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.988580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.988635 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.988691 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.990002 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.990810 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.992343 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.992691 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.992767 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.999347 (+ 6580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.999353 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.999358 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.999389 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.000951 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.001908 (+ 957us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.001914 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.001918 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.001999 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.002671 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.002678 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.002682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.002917 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.002987 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.003844 (+ 857us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.003972 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:59:01.005623 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.986853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.27.147.129:44687)
0810 18:59:00.988141 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.988150 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.988333 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.988873 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.988881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.989317 (+ 436us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.989852 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.989858 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.990929 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.990932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.992191 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.992199 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.992927 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.992933 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.993062 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.993829 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.993853 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.996338 (+ 2485us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.001084 (+ 4746us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.001089 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.001094 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.001328 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.002102 (+ 774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.002107 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.002111 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.002188 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.003586 (+ 1398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.003591 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.003677 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.004348 (+ 671us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.004670 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1216,"mutex_wait_us":378,"spinlock_wait_cycles":7040,"thread_start_us":120,"threads_started":1}
I20260810 18:59:01.006371 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.986427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:38387 (local address 127.27.147.188:32963)
0810 18:59:00.987032 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.987037 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.987553 (+ 516us) server_negotiation.cc:407] Connection header received
0810 18:59:00.988573 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.988577 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.988633 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.988694 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.990007 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.991076 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.991978 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.992362 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.992490 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.996863 (+ 4373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:00.996869 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:00.996874 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:00.996904 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:00.998625 (+ 1721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.001433 (+ 2808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.001439 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.001444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.001524 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.002294 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.002300 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.002304 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.002532 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.002608 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.005994 (+ 3386us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.006129 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":85,"threads_started":1}
I20260810 18:59:01.006916 6444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.986851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.27.147.129:38387)
0810 18:59:00.987408 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.987457 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.988330 (+ 873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.988873 (+ 543us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.988881 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.989320 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.989849 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.989858 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.991187 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.991189 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.991858 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.991865 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.992676 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.992683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.992971 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:00.993752 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:00.993891 (+ 139us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:00.996718 (+ 2827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:00.998844 (+ 2126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:00.998851 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:00.998867 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:00.999194 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.001627 (+ 2433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.001632 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.001637 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.001799 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.003055 (+ 1256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.003064 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.003366 (+ 302us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.006493 (+ 3127us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.006638 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":357,"mutex_wait_us":140,"spinlock_wait_cycles":36864,"thread_start_us":701,"threads_started":1}
I20260810 18:59:01.013883 6440 heartbeater.cc:347] Connected to a master server at 127.27.147.189:43471
I20260810 18:59:01.014204 6440 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.014956 6440 heartbeater.cc:511] Master 127.27.147.189:43471 requested a full tablet report, sending...
I20260810 18:59:01.015578 6438 heartbeater.cc:347] Connected to a master server at 127.27.147.188:32963
I20260810 18:59:01.015705 6438 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.015911 6438 heartbeater.cc:511] Master 127.27.147.188:32963 requested a full tablet report, sending...
I20260810 18:59:01.016685 5900 ts_manager.cc:194] Registered new tserver with Master: f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388340, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:01.017052 6177 ts_manager.cc:194] Registered new tserver with Master: f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
I20260810 18:59:01.021024 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.986851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:59377)
0810 18:59:00.987880 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:00.987891 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:00.988322 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:00.988872 (+ 550us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:00.988879 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:00.989283 (+ 404us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:00.989834 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.989848 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.990680 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.990683 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:00.992188 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:00.992199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.012277 (+ 20078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.012280 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.012438 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.015523 (+ 3085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:01.015540 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:01.017279 (+ 1739us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.019592 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.019595 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.019596 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.019782 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.020096 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.020099 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.020100 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.020172 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.020641 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.020643 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.020718 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.020793 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.020896 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":950,"mutex_wait_us":190,"spinlock_wait_cycles":19328,"thread_start_us":146,"threads_started":1}
I20260810 18:59:01.021311 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:00.986556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:59377 (local address 127.27.147.190:42793)
0810 18:59:00.986687 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:00.986692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:00.987992 (+ 1300us) server_negotiation.cc:407] Connection header received
0810 18:59:00.988542 (+ 550us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:00.988545 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:00.988608 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:00.988688 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:00.989985 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.990537 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.992343 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:00.992691 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:00.992772 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.017479 (+ 24707us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.017487 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.017491 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.017523 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.019451 (+ 1928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.019894 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.019900 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.019905 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.020007 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.020266 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.020272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.020276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.020543 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.020647 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.020893 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.021038 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 18:59:01.022245 6439 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42793
I20260810 18:59:01.022346 6439 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.022518 6439 heartbeater.cc:511] Master 127.27.147.190:42793 requested a full tablet report, sending...
I20260810 18:59:01.022539 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:44075
--local_ip_for_outbound_sockets=127.27.147.130
--tserver_master_addrs=127.27.147.190:42793
--webserver_port=46431
--webserver_interface=127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab
--principal=oryx/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:59:01.023322 5025 ts_manager.cc:194] Re-registered known tserver with Master: f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
I20260810 18:59:01.023903 5025 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:59377
W20260810 18:59:01.136291 6447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:01.136541 6447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:01.136584 6447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:01.140734 6447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:01.140784 6447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:01.140837 6447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:59:01.146378 6447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab
--principal=oryx/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:44075
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=46431
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:01.147338 6447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:01.148063 5694 heartbeater.cc:503] Master 127.27.147.190:42793 was elected leader, sending a full tablet report...
I20260810 18:59:01.148425 6447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:01.158593 6463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:01.158538 6457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:01.158684 6447 server_base.cc:1034] running on GCE node
W20260810 18:59:01.158499 6456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:01.159204 6447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:01.159725 6447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:01.160909 6447 hybrid_clock.cc:648] HybridClock initialized: now 1786388341160894 us; error 57 us; skew 500 ppm
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:01.163496 6447 init.cc:377] Logged in from keytab as oryx/127.27.147.130@KRBTEST.COM (short username oryx)
I20260810 18:59:01.164223 6447 webserver.cc:460] Webserver started at http://127.27.147.130:46431/ using document root <none> and password file <none>
I20260810 18:59:01.164697 6447 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:01.164768 6447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:01.168232 6447 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.003s
I20260810 18:59:01.170109 6473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:01.171314 6447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:59:01.171440 6447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "c6ab1c37242045b0806525c8ecbb76c3"
format_stamp: "Formatted at 2026-08-10 18:58:59 on dist-test-slave-znh6"
server_key: "aeca7b11c87fa7c5d0eb125b2cc2daee"
server_key_iv: "c2145e349b125b4e823923df26fab203"
server_key_version: "encryptionkey@0"
I20260810 18:59:01.171903 6447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:01.196017 6447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:01.198676 6447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:01.198879 6447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:01.199453 6447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:01.200223 6447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:01.200269 6447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:01.200306 6447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:01.200331 6447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:01.209120 6447 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:44075
I20260810 18:59:01.209146 6643 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:44075 every 8 connection(s)
I20260810 18:59:01.210074 6447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:59:01.219883 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 6447
I20260810 18:59:01.220081 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 5513
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.188@KRBTEST.COM
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.189@KRBTEST.COM
I20260810 18:59:01.227738 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.211924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:49027 (local address 127.27.147.190:42793)
0810 18:59:01.212086 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.212090 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.213781 (+ 1691us) server_negotiation.cc:407] Connection header received
0810 18:59:01.214392 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.214399 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.214463 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.214558 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.215968 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.216789 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218250 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.218585 (+ 335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218685 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.222439 (+ 3754us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.222445 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.222462 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.222491 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.225034 (+ 2543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.225629 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.225635 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.225640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.225723 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.226079 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.226085 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.226089 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.226309 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.226380 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.227287 (+ 907us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.227456 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 18:59:01.228595 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.212436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:45443 (local address 127.27.147.188:32963)
0810 18:59:01.212596 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.212602 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.213524 (+ 922us) server_negotiation.cc:407] Connection header received
0810 18:59:01.214503 (+ 979us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.214507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.214561 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.214628 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.215989 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.216532 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218063 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.218424 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218536 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.223039 (+ 4503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.223044 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.223047 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.223076 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.225176 (+ 2100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.227226 (+ 2050us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.227231 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.227235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.227312 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.227573 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.227578 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.227582 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.227793 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.227868 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.228252 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.228392 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 18:59:01.229662 6654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.212329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:49027)
0810 18:59:01.213707 (+ 1378us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.213720 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.214187 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.214745 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.214751 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.215253 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:01.215820 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.215833 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.217334 (+ 1501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.217338 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.218140 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.218147 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.218700 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218703 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.218871 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.219545 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:01.219563 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:01.222263 (+ 2700us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.225161 (+ 2898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.225168 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.225185 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.225522 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.225822 (+ 300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.225827 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.225831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.225985 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.226481 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.226490 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.226925 (+ 435us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.227939 (+ 1014us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.228402 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1295,"mutex_wait_us":1034,"thread_start_us":76,"threads_started":1}
I20260810 18:59:01.229908 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.212336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.27.147.130:45443)
0810 18:59:01.213430 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.213444 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.214183 (+ 739us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.214754 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.214757 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.215255 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:01.215821 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.215836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.216679 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.216683 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.217936 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.217947 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.218536 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218540 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.218660 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.219331 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:01.219346 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:01.222781 (+ 3435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.226882 (+ 4101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.226887 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.226892 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.227123 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.227400 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.227405 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.227409 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.227485 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.227906 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.227911 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.228019 (+ 108us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.228109 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.228902 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1023,"mutex_wait_us":787,"spinlock_wait_cycles":51584,"thread_start_us":192,"threads_started":1}
I20260810 18:59:01.231043 6644 heartbeater.cc:347] Connected to a master server at 127.27.147.188:32963
I20260810 18:59:01.231364 6644 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.231992 6644 heartbeater.cc:511] Master 127.27.147.188:32963 requested a full tablet report, sending...
I20260810 18:59:01.232800 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.211976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:46605 (local address 127.27.147.189:43471)
0810 18:59:01.212100 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.212104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.213207 (+ 1103us) server_negotiation.cc:407] Connection header received
0810 18:59:01.214435 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.214438 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.214501 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.214558 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.215999 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.216789 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.217956 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.218282 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218381 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.223373 (+ 4992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.223379 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.223385 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.223417 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.226759 (+ 3342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.231405 (+ 4646us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.231411 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.231415 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.231500 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.231816 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.231821 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.231826 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.232037 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.232104 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.232510 (+ 406us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.232635 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:59:01.233250 6177 ts_manager.cc:194] Registered new tserver with Master: c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130:44075)
I20260810 18:59:01.233654 6650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.212329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.27.147.130:46605)
0810 18:59:01.213023 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.213091 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.214183 (+ 1092us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.214934 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.214938 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.215256 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:01.215821 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.215836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.216901 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.216905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.217821 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.217831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.218436 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.218440 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.218645 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.219399 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:01.219421 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:01.223185 (+ 3764us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.231025 (+ 7840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.231030 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.231035 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.231301 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.231609 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.231615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.231620 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.231710 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.232196 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.232198 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.232286 (+ 88us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.232427 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.233474 (+ 1047us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"mutex_wait_us":142,"spinlock_wait_cycles":31488,"thread_start_us":770,"threads_started":1}
I20260810 18:59:01.234360 6648 heartbeater.cc:347] Connected to a master server at 127.27.147.189:43471
I20260810 18:59:01.234455 6648 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.234627 6648 heartbeater.cc:511] Master 127.27.147.189:43471 requested a full tablet report, sending...
I20260810 18:59:01.234881 6647 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42793
I20260810 18:59:01.234972 6647 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.235144 6647 heartbeater.cc:511] Master 127.27.147.190:42793 requested a full tablet report, sending...
I20260810 18:59:01.235433 5900 ts_manager.cc:194] Registered new tserver with Master: c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130:44075)
I20260810 18:59:01.235790 5025 ts_manager.cc:194] Re-registered known tserver with Master: c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130:44075)
I20260810 18:59:01.236348 5025 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:49027
I20260810 18:59:01.241621 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:38435
--local_ip_for_outbound_sockets=127.27.147.131
--tserver_master_addrs=127.27.147.190:42793
--webserver_port=39731
--webserver_interface=127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab
--principal=oryx/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:01.357764 6655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:01.358026 6655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:01.358111 6655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:01.362013 6655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:01.362087 6655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:01.362169 6655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:01.367292 6655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab
--principal=oryx/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:38435
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=39731
--tserver_master_addrs=127.27.147.190:42793,127.27.147.189:43471,127.27.147.188:32963
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:01.368258 6655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:01.369227 6655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:01.379266 6672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:01.379276 6670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:01.379300 6669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:01.379508 6655 server_base.cc:1034] running on GCE node
I20260810 18:59:01.379937 6655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:01.380564 6655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:01.381750 6655 hybrid_clock.cc:648] HybridClock initialized: now 1786388341381733 us; error 30 us; skew 500 ppm
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:01.384575 6655 init.cc:377] Logged in from keytab as oryx/127.27.147.131@KRBTEST.COM (short username oryx)
I20260810 18:59:01.385594 6655 webserver.cc:460] Webserver started at http://127.27.147.131:39731/ using document root <none> and password file <none>
I20260810 18:59:01.386226 6655 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:01.386348 6655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:01.388643 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.384258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.0.0.1:58782)
0810 18:59:01.384523 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.384574 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.384702 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.385097 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.385100 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.385121 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:01.385214 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.385219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.386229 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.386232 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.387420 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.387431 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.388301 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.388304 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.388393 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.388415 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.388468 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.388527 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":75,"threads_started":1}
I20260810 18:59:01.388695 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.384342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.27.147.188:32963)
0810 18:59:01.384492 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.384497 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.384665 (+ 168us) server_negotiation.cc:407] Connection header received
0810 18:59:01.384889 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.384892 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.384961 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.385060 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:01.385319 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.386120 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.387545 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.388184 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.388311 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.388467 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.388507 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.388575 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 18:59:01.389365 6204 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260810 18:59:01.389753 6242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b007540de45341a0a6c5ef083c571502. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:01.389921 6242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: This master's current role is: LEARNER
I20260810 18:59:01.390947 6655 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:59:01.393085 6681 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:01.394557 6655 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:01.394688 6655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "2a533e704c5e4f4eaf7851abf9828d86"
format_stamp: "Formatted at 2026-08-10 18:59:00 on dist-test-slave-znh6"
server_key: "f93ed64f7cc3322142ff44f0bb5758e5"
server_key_iv: "e8a6cc6178b2dcbdd3c0d6546547b312"
server_key_version: "encryptionkey@0"
I20260810 18:59:01.395321 6655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:01.411566 6655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:01.414620 6655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:01.414808 6655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:01.415395 6655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:01.416237 6655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:01.416309 6655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:01.416378 6655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:01.416425 6655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:01.424919 6655 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:38435
I20260810 18:59:01.424943 6853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:38435 every 8 connection(s)
I20260810 18:59:01.425827 6655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:01.429170 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 6655
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.188@KRBTEST.COM
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
Aug 10 18:59:01 dist-test-slave-znh6 krb5kdc[4957](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388341, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.189@KRBTEST.COM
I20260810 18:59:01.444298 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.428131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:47375 (local address 127.27.147.188:32963)
0810 18:59:01.428244 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.428249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.429240 (+ 991us) server_negotiation.cc:407] Connection header received
0810 18:59:01.430102 (+ 862us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.430108 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.430164 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.430237 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.432123 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.432908 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.434543 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.434895 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.434990 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.438487 (+ 3497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.438491 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.438493 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.438518 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.440365 (+ 1847us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.442343 (+ 1978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.442347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.442349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.442434 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.442927 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.442934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.442937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.443172 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.443271 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.443985 (+ 714us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.444114 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:59:01.444339 6878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.432454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.27.147.190:42793)
0810 18:59:01.433674 (+ 1220us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.433678 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.433691 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:01.433726 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.433728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.433769 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.434412 (+ 643us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.435756 (+ 1344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.436527 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.437752 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.438084 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.438179 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.440024 (+ 1845us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.440029 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.440032 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.440057 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.441876 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.442413 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.442419 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.442421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.442512 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.442928 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.442933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.442936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.443184 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.443346 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.443971 (+ 625us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.444104 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1145,"thread_start_us":76,"threads_started":1}
I20260810 18:59:01.446208 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.427947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.27.147.131:47375)
0810 18:59:01.429140 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.429154 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.429778 (+ 624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.430446 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.430455 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.431030 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:01.431983 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.431993 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.433031 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.433034 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.434404 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.434414 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.435151 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.435155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.435400 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.436245 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:01.436273 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:01.438299 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.441821 (+ 3522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.441827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.441841 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.442220 (+ 379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.442549 (+ 329us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.442552 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.442554 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.442731 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.443354 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.443360 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.443658 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.444468 (+ 810us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.444824 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1115,"mutex_wait_us":548,"spinlock_wait_cycles":33024,"thread_start_us":159,"threads_started":1}
I20260810 18:59:01.447494 6854 heartbeater.cc:347] Connected to a master server at 127.27.147.188:32963
I20260810 18:59:01.447600 5082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.427526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:60201 (local address 127.27.147.190:42793)
0810 18:59:01.427646 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.427650 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.428806 (+ 1156us) server_negotiation.cc:407] Connection header received
0810 18:59:01.430016 (+ 1210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.430019 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.430078 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.430131 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.432655 (+ 2524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.433383 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.434541 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.434891 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.434996 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.442562 (+ 7566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.442565 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.442568 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.442595 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.445424 (+ 2829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.445909 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.445914 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.445916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.446009 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.446601 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.446605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.446607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.446852 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.446932 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.447324 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.447447 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:59:01.447724 6854 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.448275 6854 heartbeater.cc:511] Master 127.27.147.188:32963 requested a full tablet report, sending...
I20260810 18:59:01.448315 6862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.427947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:60201)
0810 18:59:01.428663 (+ 716us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.428712 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.429778 (+ 1066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.430446 (+ 668us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.430461 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.431078 (+ 617us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:01.431777 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.431802 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.433522 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.433525 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.434429 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.434441 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.435151 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.435155 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.435785 (+ 630us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.436572 (+ 787us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:01.436589 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:01.442405 (+ 5816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.445553 (+ 3148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.445556 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.445558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.445790 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.446149 (+ 359us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.446153 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.446156 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.446251 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.447013 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.447016 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.447114 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.447984 (+ 870us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.448126 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":534,"mutex_wait_us":605,"spinlock_wait_cycles":1792,"thread_start_us":699,"threads_started":1}
I20260810 18:59:01.448797 6877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.432186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56758 (local address 127.27.147.189:43471)
0810 18:59:01.433272 (+ 1086us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.433276 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.433289 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:01.433323 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.433326 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.433372 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.433460 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.436894 (+ 3434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.437657 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.439220 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.439881 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.439979 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.444106 (+ 4127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.444110 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.444113 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.444138 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.446753 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.447241 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.447244 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.447246 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.447352 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.447659 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.447663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.447664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.447867 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.447983 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.448521 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.448638 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1008,"thread_start_us":89,"threads_started":1}
I20260810 18:59:01.449530 6177 ts_manager.cc:194] Registered new tserver with Master: 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131:38435)
I20260810 18:59:01.450162 6858 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42793
I20260810 18:59:01.450244 6858 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.450441 6858 heartbeater.cc:511] Master 127.27.147.190:42793 requested a full tablet report, sending...
I20260810 18:59:01.447685 5025 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:59:01.451387 5024 ts_manager.cc:194] Re-registered known tserver with Master: 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131:38435)
W20260810 18:59:01.451732 5025 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:01.452165 5024 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:60201
I20260810 18:59:01.452747 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.428133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:33407 (local address 127.27.147.189:43471)
0810 18:59:01.428244 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.428249 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.429543 (+ 1294us) server_negotiation.cc:407] Connection header received
0810 18:59:01.430051 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.430054 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.430120 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.430184 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.431962 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.432714 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.434705 (+ 1991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.435117 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.435203 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.443349 (+ 8146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.443353 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.443355 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.443392 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.450171 (+ 6779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.451323 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.451327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.451328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.451415 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.451815 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.451818 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.451820 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.452018 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.452106 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.452500 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.452607 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 18:59:01.453264 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.427955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.27.147.131:33407)
0810 18:59:01.429439 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.429452 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.429791 (+ 339us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.430448 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.430456 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.431029 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:01.431780 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.431801 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.432905 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.432908 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.434581 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.434588 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.435396 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.435401 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.435520 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.436414 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:01.436429 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:01.443177 (+ 6748us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:01.450932 (+ 7755us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.450935 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.450937 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.451177 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.451576 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:01.451579 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:01.451580 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:01.451666 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:01.452182 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:01.452186 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:01.452312 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.452940 (+ 628us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.453106 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"mutex_wait_us":382,"spinlock_wait_cycles":24832,"thread_start_us":140,"threads_started":1}
I20260810 18:59:01.453181 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.432657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58788 (local address 127.27.147.188:32963)
0810 18:59:01.433981 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.433986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.433997 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:59:01.434031 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.434033 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.434073 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.434153 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.437082 (+ 2929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.437963 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.439193 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.439596 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.439706 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.441837 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.441840 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.441843 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.441865 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.450988 (+ 9123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.451489 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.451492 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.451494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.451577 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.451886 (+ 309us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.451889 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.451891 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.452080 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.452199 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.452823 (+ 624us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.452952 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1252,"thread_start_us":73,"threads_started":1}
I20260810 18:59:01.453867 6859 heartbeater.cc:347] Connected to a master server at 127.27.147.189:43471
I20260810 18:59:01.453943 6859 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:01.454058 6859 heartbeater.cc:511] Master 127.27.147.189:43471 requested a full tablet report, sending...
I20260810 18:59:01.454698 5900 ts_manager.cc:194] Registered new tserver with Master: 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131:38435)
I20260810 18:59:01.455737 6881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: NON_VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: true } } }
I20260810 18:59:01.455860 6881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:01.459028 6881 raft_consensus.cc:1050] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: attempting to promote NON_VOTER 8c2b5bc1db18407da8e34c713db85f36 to VOTER
I20260810 18:59:01.459410 6881 consensus_queue.cc:237] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:01.460515 6204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 LEARNER]: Refusing update from remote peer b007540de45341a0a6c5ef083c571502: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:59:01.460515 5918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Refusing update from remote peer b007540de45341a0a6c5ef083c571502: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260810 18:59:01.460878 6656 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:59:01.460996 6882 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260810 18:59:01.462607 6881 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.463095 5918 raft_consensus.cc:2932] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.464184 6881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.464298 6881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:01.464001 6204 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.464464 6881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.464564 6881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:01.464685 6881 sys_catalog.cc:455] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.464777 6881 sys_catalog.cc:458] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:01.465565 6898 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.465725 6898 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:59:01.466439 6242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "b007540de45341a0a6c5ef083c571502" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:01.466553 6242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:59:01.473438 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.468206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:38435 (local address 127.0.0.1:53304)
0810 18:59:01.469042 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.469056 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.469158 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.469657 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.469660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.469675 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:01.469773 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.469779 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.470799 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.470802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.472010 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.472021 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.473071 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.473074 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.473163 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.473187 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.473237 (+ 50us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.473307 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":774}
I20260810 18:59:01.474056 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.468324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53304 (local address 127.27.147.131:38435)
0810 18:59:01.468705 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.468719 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.469178 (+ 459us) server_negotiation.cc:407] Connection header received
0810 18:59:01.469263 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.469269 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.469490 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.469694 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:01.469887 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.470679 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.472117 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.472925 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.473633 (+ 708us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.473764 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.473817 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.473907 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":147,"threads_started":1}
I20260810 18:59:01.475391 6904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.468603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44075 (local address 127.0.0.1:59272)
0810 18:59:01.469431 (+ 828us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.469446 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.469535 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.470329 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.470332 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.470350 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:01.470434 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.470440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.471799 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.471802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.472989 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.473002 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.474988 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.474992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.475082 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.475104 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.475202 (+ 98us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.475272 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":105,"threads_started":1}
I20260810 18:59:01.475852 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.469022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59272 (local address 127.27.147.130:44075)
0810 18:59:01.469901 (+ 879us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.469913 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.469935 (+ 22us) server_negotiation.cc:407] Connection header received
0810 18:59:01.470003 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.470006 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.470204 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.470348 (+ 144us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:01.470570 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.471648 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.473122 (+ 1474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.473944 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.474068 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.474917 (+ 849us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.475608 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.475708 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":766,"thread_start_us":111,"threads_started":1}
I20260810 18:59:01.477600 6900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.468768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37149 (local address 127.0.0.1:53752)
0810 18:59:01.469495 (+ 727us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.469517 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.469613 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.471122 (+ 1509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.471125 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.471144 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:01.471228 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.471233 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.473255 (+ 2022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.473260 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.475837 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.475852 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.477240 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.477243 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.477327 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.477349 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.477421 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.477484 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":109,"threads_started":1}
I20260810 18:59:01.477151 6751 tablet_service.cc:1467] Processing CreateTablet for tablet 1b35a8d981b14ff4a5373ffa5aec36e3 (DEFAULT_TABLE table=test-table [id=dfef144944d0457da8385921a7f8aba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:01.478047 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.468951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53752 (local address 127.27.147.129:37149)
0810 18:59:01.470638 (+ 1687us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.470652 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.470676 (+ 24us) server_negotiation.cc:407] Connection header received
0810 18:59:01.470754 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.470762 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.470981 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.471127 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:01.471355 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.473119 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.476125 (+ 3006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.476927 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.477048 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.477168 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.477776 (+ 608us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.477891 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1564,"thread_start_us":198,"threads_started":1}
I20260810 18:59:01.478312 6751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b35a8d981b14ff4a5373ffa5aec36e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:01.478147 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 1b35a8d981b14ff4a5373ffa5aec36e3 (DEFAULT_TABLE table=test-table [id=dfef144944d0457da8385921a7f8aba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:01.479146 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b35a8d981b14ff4a5373ffa5aec36e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:01.480592 6351 tablet_service.cc:1467] Processing CreateTablet for tablet 1b35a8d981b14ff4a5373ffa5aec36e3 (DEFAULT_TABLE table=test-table [id=dfef144944d0457da8385921a7f8aba9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:01.481503 6351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1b35a8d981b14ff4a5373ffa5aec36e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:01.485528 6910 tablet_bootstrap.cc:492] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Bootstrap starting.
I20260810 18:59:01.487126 6911 tablet_bootstrap.cc:492] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Bootstrap starting.
I20260810 18:59:01.488221 6910 tablet_bootstrap.cc:654] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:01.489025 6912 tablet_bootstrap.cc:492] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Bootstrap starting.
I20260810 18:59:01.489118 6910 log.cc:824] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:01.489300 6911 tablet_bootstrap.cc:654] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:01.490087 6911 log.cc:824] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:01.491067 6912 tablet_bootstrap.cc:654] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:01.491703 6912 log.cc:824] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:01.492025 6911 tablet_bootstrap.cc:492] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: No bootstrap required, opened a new log
I20260810 18:59:01.492223 6911 ts_tablet_manager.cc:1397] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:59:01.493664 6912 tablet_bootstrap.cc:492] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: No bootstrap required, opened a new log
I20260810 18:59:01.493907 6912 ts_tablet_manager.cc:1397] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:59:01.495909 6910 tablet_bootstrap.cc:492] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: No bootstrap required, opened a new log
I20260810 18:59:01.496098 6910 ts_tablet_manager.cc:1397] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20260810 18:59:01.497128 6911 raft_consensus.cc:348] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.497428 6911 raft_consensus.cc:374] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:01.497489 6911 raft_consensus.cc:729] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ab1c37242045b0806525c8ecbb76c3, State: Initialized, Role: FOLLOWER
I20260810 18:59:01.497889 6912 raft_consensus.cc:348] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.498032 6911 consensus_queue.cc:260] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.498153 6912 raft_consensus.cc:374] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:01.498207 6912 raft_consensus.cc:729] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34121eec10941dfbb6f650604c8656f, State: Initialized, Role: FOLLOWER
I20260810 18:59:01.498667 6912 consensus_queue.cc:260] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.499294 6911 ts_tablet_manager.cc:1428] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Time spent starting tablet: real 0.007s user 0.007s sys 0.001s
I20260810 18:59:01.499487 6647 heartbeater.cc:503] Master 127.27.147.190:42793 was elected leader, sending a full tablet report...
I20260810 18:59:01.500093 6910 raft_consensus.cc:348] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.500327 6910 raft_consensus.cc:374] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:01.500379 6910 raft_consensus.cc:729] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a533e704c5e4f4eaf7851abf9828d86, State: Initialized, Role: FOLLOWER
I20260810 18:59:01.500765 6910 consensus_queue.cc:260] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.501690 6910 ts_tablet_manager.cc:1428] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:59:01.501979 6858 heartbeater.cc:503] Master 127.27.147.190:42793 was elected leader, sending a full tablet report...
I20260810 18:59:01.503341 6912 ts_tablet_manager.cc:1428] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260810 18:59:01.504586 6439 heartbeater.cc:503] Master 127.27.147.190:42793 was elected leader, sending a full tablet report...
W20260810 18:59:01.677390 6860 tablet.cc:2369] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:59:01.711792 6649 tablet.cc:2369] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:59:01.736720 6441 tablet.cc:2369] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:59:01.742791 6925 raft_consensus.cc:482] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:01.743017 6925 raft_consensus.cc:504] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.744097 6925 leader_election.cc:288] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130:44075), f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
I20260810 18:59:01.748126 6861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.744355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44075 (local address 127.27.147.131:33355)
0810 18:59:01.744578 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.744594 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.744694 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.745015 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.745021 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.745038 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:01.745189 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.745198 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.746136 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.746141 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.747052 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.747062 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.747693 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.747697 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.747812 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.747831 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.747916 (+ 85us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.747990 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":38}
I20260810 18:59:01.748126 6905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.744545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:33355 (local address 127.27.147.130:44075)
0810 18:59:01.744683 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.744688 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.744706 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:59:01.744825 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.744829 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.744892 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.744992 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:01.745335 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.746015 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.747177 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.747573 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.747700 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.747753 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.747935 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.748012 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:59:01.748373 6906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.744556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:54165 (local address 127.27.147.129:37149)
0810 18:59:01.744672 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.744676 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.745032 (+ 356us) server_negotiation.cc:407] Connection header received
0810 18:59:01.745089 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.745092 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.745142 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.745213 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:01.745634 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.746340 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.747437 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.747824 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.747950 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.748013 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.748185 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.748264 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:59:01.748369 6863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.744355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37149 (local address 127.27.147.131:54165)
0810 18:59:01.744851 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:01.744863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:01.744935 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:01.745328 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:01.745331 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:01.745345 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:01.745500 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.745507 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.746452 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.746457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.747312 (+ 855us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:01.747322 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.747944 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.747948 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:01.748055 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.748082 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 18:59:01.748166 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 18:59:01.748241 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":437,"mutex_wait_us":332}
I20260810 18:59:01.749101 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b35a8d981b14ff4a5373ffa5aec36e3" candidate_uuid: "2a533e704c5e4f4eaf7851abf9828d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ab1c37242045b0806525c8ecbb76c3" is_pre_election: true
I20260810 18:59:01.749115 6375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b35a8d981b14ff4a5373ffa5aec36e3" candidate_uuid: "2a533e704c5e4f4eaf7851abf9828d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34121eec10941dfbb6f650604c8656f" is_pre_election: true
I20260810 18:59:01.749418 6375 raft_consensus.cc:2454] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a533e704c5e4f4eaf7851abf9828d86 in term 0.
I20260810 18:59:01.749418 6577 raft_consensus.cc:2454] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a533e704c5e4f4eaf7851abf9828d86 in term 0.
I20260810 18:59:01.749925 6684 leader_election.cc:302] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2a533e704c5e4f4eaf7851abf9828d86, c6ab1c37242045b0806525c8ecbb76c3, f34121eec10941dfbb6f650604c8656f; no voters:
I20260810 18:59:01.750212 6925 raft_consensus.cc:2781] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:01.750298 6925 raft_consensus.cc:482] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:01.750350 6925 raft_consensus.cc:3037] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:01.751564 6925 raft_consensus.cc:504] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.751881 6925 leader_election.cc:288] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [CANDIDATE]: Term 1 election: Requested vote from peers c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130:44075), f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
I20260810 18:59:01.752170 6577 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b35a8d981b14ff4a5373ffa5aec36e3" candidate_uuid: "2a533e704c5e4f4eaf7851abf9828d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ab1c37242045b0806525c8ecbb76c3"
I20260810 18:59:01.752264 6375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1b35a8d981b14ff4a5373ffa5aec36e3" candidate_uuid: "2a533e704c5e4f4eaf7851abf9828d86" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34121eec10941dfbb6f650604c8656f"
I20260810 18:59:01.752331 6577 raft_consensus.cc:3037] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:01.752413 6375 raft_consensus.cc:3037] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:01.753361 6577 raft_consensus.cc:2454] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a533e704c5e4f4eaf7851abf9828d86 in term 1.
I20260810 18:59:01.753464 6375 raft_consensus.cc:2454] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a533e704c5e4f4eaf7851abf9828d86 in term 1.
I20260810 18:59:01.753665 6683 leader_election.cc:302] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a533e704c5e4f4eaf7851abf9828d86, c6ab1c37242045b0806525c8ecbb76c3; no voters:
I20260810 18:59:01.753813 6925 raft_consensus.cc:2781] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:01.754040 6925 raft_consensus.cc:686] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 LEADER]: Becoming Leader. State: Replica: 2a533e704c5e4f4eaf7851abf9828d86, State: Running, Role: LEADER
I20260810 18:59:01.754380 6925 consensus_queue.cc:237] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:01.757812 5025 catalog_manager.cc:5495] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131). New cstate: current_term: 1 leader_uuid: "2a533e704c5e4f4eaf7851abf9828d86" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } health_report { overall_health: UNKNOWN } } }
I20260810 18:59:01.839536 6899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.836355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53306 (local address 127.27.147.131:38435)
0810 18:59:01.836478 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.836481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.836496 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:59:01.836572 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.836574 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.836628 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.836716 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:01.836929 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.837444 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.838233 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.838457 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.838534 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.838784 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:01.839200 (+ 416us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:01.839318 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.839362 (+ 44us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.839410 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:59:01.845222 6375 raft_consensus.cc:1261] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 1 FOLLOWER]: Refusing update from remote peer 2a533e704c5e4f4eaf7851abf9828d86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:01.845221 6577 raft_consensus.cc:1261] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 1 FOLLOWER]: Refusing update from remote peer 2a533e704c5e4f4eaf7851abf9828d86: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:01.845983 6926 consensus_queue.cc:1035] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:01.846201 6925 consensus_queue.cc:1035] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:01.855141 6933 mvcc.cc:205] Tried to move back new op lower bound from 7317046648190844928 to 7317046647828676608. Current Snapshot: MvccSnapshot[applied={T|T < 7317046648190844928}]
I20260810 18:59:01.860144 6931 mvcc.cc:205] Tried to move back new op lower bound from 7317046648190844928 to 7317046647828676608. Current Snapshot: MvccSnapshot[applied={T|T < 7317046648190844928}]
I20260810 18:59:01.864574 5025 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:01.864738 5025 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34542:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:01.868903 5025 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b007540de45341a0a6c5ef083c571502: Sending DeleteTablet for 3 replicas of tablet 1b35a8d981b14ff4a5373ffa5aec36e3
I20260810 18:59:01.869757 6351 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b35a8d981b14ff4a5373ffa5aec36e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:01 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:53752
I20260810 18:59:01.869742 6547 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b35a8d981b14ff4a5373ffa5aec36e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:01 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:59272
I20260810 18:59:01.870205 6944 tablet_replica.cc:330] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: stopping tablet replica
I20260810 18:59:01.870743 6751 tablet_service.cc:1514] Processing DeleteTablet for tablet 1b35a8d981b14ff4a5373ffa5aec36e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:01 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:53304
I20260810 18:59:01.871096 6943 tablet_replica.cc:330] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: stopping tablet replica
I20260810 18:59:01.870550 6944 raft_consensus.cc:2229] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:01.871486 6943 raft_consensus.cc:2229] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:01.871570 6944 raft_consensus.cc:2258] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:01.871805 6943 raft_consensus.cc:2258] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:01.871948 6947 tablet_replica.cc:330] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: stopping tablet replica
I20260810 18:59:01.872213 6947 raft_consensus.cc:2229] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:59:01.872556 6947 raft_consensus.cc:2258] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:01.873217 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 4975
I20260810 18:59:01.883690 6944 ts_tablet_manager.cc:1905] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:01.887344 6944 ts_tablet_manager.cc:1918] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:01.888489 6943 ts_tablet_manager.cc:1905] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:01.888686 6944 log.cc:1196] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/1b35a8d981b14ff4a5373ffa5aec36e3
I20260810 18:59:01.889132 6944 ts_tablet_manager.cc:1939] T 1b35a8d981b14ff4a5373ffa5aec36e3 P f34121eec10941dfbb6f650604c8656f: Deleting consensus metadata
I20260810 18:59:01.891572 6947 ts_tablet_manager.cc:1905] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 18:59:01.892889 6294 connection.cc:441] server connection from 127.0.0.1:53752 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53752 (request call id 1) could send its response
I20260810 18:59:01.893600 6943 ts_tablet_manager.cc:1918] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:01.893780 6943 log.cc:1196] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/1b35a8d981b14ff4a5373ffa5aec36e3
I20260810 18:59:01.894289 6943 ts_tablet_manager.cc:1939] T 1b35a8d981b14ff4a5373ffa5aec36e3 P c6ab1c37242045b0806525c8ecbb76c3: Deleting consensus metadata
W20260810 18:59:01.895486 6479 connection.cc:441] server connection from 127.0.0.1:59272 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59272 (request call id 1) could send its response
I20260810 18:59:01.895572 6947 ts_tablet_manager.cc:1918] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:01.895692 6947 log.cc:1196] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/1b35a8d981b14ff4a5373ffa5aec36e3
I20260810 18:59:01.896051 6947 ts_tablet_manager.cc:1939] T 1b35a8d981b14ff4a5373ffa5aec36e3 P 2a533e704c5e4f4eaf7851abf9828d86: Deleting consensus metadata
W20260810 18:59:01.897083 6684 connection.cc:441] server connection from 127.0.0.1:53304 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:53304 (request call id 1) could send its response
I20260810 18:59:01.901878 6088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.872250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56770 (local address 127.27.147.189:43471)
0810 18:59:01.872589 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.872592 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.891091 (+ 18499us) server_negotiation.cc:407] Connection header received
0810 18:59:01.891303 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.891307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.891360 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.891413 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.892459 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.893282 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.894612 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.894961 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.895041 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.897413 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.897417 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.897420 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.897445 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.899856 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.900367 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.900370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.900372 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.900455 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.900809 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.900813 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.900816 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.901051 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.901174 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.901560 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.901687 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258}
I20260810 18:59:01.902935 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.891714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58796 (local address 127.27.147.188:32963)
0810 18:59:01.891836 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.891841 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.893482 (+ 1641us) server_negotiation.cc:407] Connection header received
0810 18:59:01.893678 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.893684 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.893735 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.893789 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.894905 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.895644 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.897008 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.897347 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.897427 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.898992 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.898996 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.898999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.899024 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.901033 (+ 2009us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.901534 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.901537 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.901539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.901628 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.901919 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.901923 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.901924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.902156 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.902261 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.902607 (+ 346us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.902724 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:59:01.905501 6877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.892816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56786 (local address 127.27.147.189:43471)
0810 18:59:01.892933 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.892937 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.896304 (+ 3367us) server_negotiation.cc:407] Connection header received
0810 18:59:01.896462 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.896476 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.896520 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.896567 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.897494 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.898321 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.899741 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.900102 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.900202 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.902009 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.902014 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.902017 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.902044 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.903754 (+ 1710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.904292 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.904296 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.904298 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.904384 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.904681 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.904685 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.904687 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.904880 (+ 193us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.905009 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.905239 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.905358 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 18:59:01.911248 6879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:01.871987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58792 (local address 127.27.147.188:32963)
0810 18:59:01.872101 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:01.872108 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:01.903441 (+ 31333us) server_negotiation.cc:407] Connection header received
0810 18:59:01.903621 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:01.903626 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:01.903677 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:01.903729 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:01.904801 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.905520 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.906651 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:01.906961 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:01.907067 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:01.908290 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:01.908294 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:01.908296 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:01.908320 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:01.909632 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.910036 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.910040 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.910042 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.910125 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:01.910392 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:01.910396 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:01.910398 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:01.910563 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:01.910691 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:01.910972 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 18:59:01.911082 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260810 18:59:02.857947 6961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:02.857530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:57185)
0810 18:59:02.857766 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:02.857851 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":70,"threads_started":1}
W20260810 18:59:02.858294 6647 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:42793 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:02.860502 6962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:02.859689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:34013)
0810 18:59:02.860306 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:02.860396 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":68,"threads_started":1}
I20260810 18:59:02.860735 6963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:02.860101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:44767)
0810 18:59:02.860538 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:02.860631 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":66,"threads_started":1}
W20260810 18:59:02.860878 6858 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:42793 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
W20260810 18:59:02.860967 6439 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:42793 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:03.369618 6965 raft_consensus.cc:482] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b007540de45341a0a6c5ef083c571502)
I20260810 18:59:03.369808 6965 raft_consensus.cc:504] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:03.370273 6966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader b007540de45341a0a6c5ef083c571502)
I20260810 18:59:03.370479 6966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:03.371059 6965 leader_election.cc:288] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793), 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188:32963)
I20260810 18:59:03.371387 6966 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793), f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189:43471)
I20260810 18:59:03.371694 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.371154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34946)
0810 18:59:03.371385 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.371576 (+ 191us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":57,"threads_started":1}
I20260810 18:59:03.371945 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.371472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34952)
0810 18:59:03.371786 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.371863 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":238,"thread_start_us":75,"threads_started":1}
W20260810 18:59:03.371950 5881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
W20260810 18:59:03.372097 6169 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:03.373031 6970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.372779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34956)
0810 18:59:03.372916 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.372976 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 18:59:03.373031 6967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.372802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34970)
0810 18:59:03.372925 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.372967 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260810 18:59:03.373144 6169 leader_election.cc:334] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793): Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
W20260810 18:59:03.373229 5881 leader_election.cc:334] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793): Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:03.376479 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.371283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:32963 (local address 127.0.0.1:58812)
0810 18:59:03.372186 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.372202 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:03.372295 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:03.372662 (+ 367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:03.372665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:03.372680 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:03.372773 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:03.372782 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.373792 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.373796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:03.375134 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:03.375146 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.376010 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.376013 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:03.376168 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:03.376196 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:59:03.376270 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 18:59:03.376334 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":833,"mutex_wait_us":630,"thread_start_us":67,"threads_started":1}
I20260810 18:59:03.376533 6973 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.371778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58812 (local address 127.27.147.188:32963)
0810 18:59:03.372148 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:03.372153 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:03.372376 (+ 223us) server_negotiation.cc:407] Connection header received
0810 18:59:03.372424 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:03.372429 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:03.372487 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:03.372560 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:03.372892 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.373671 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.375260 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.375889 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.376017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:03.376072 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:03.376324 (+ 252us) server_negotiation.cc:299] Negotiation successful
0810 18:59:03.376402 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":55,"threads_started":1}
I20260810 18:59:03.376557 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.371596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:43471 (local address 127.0.0.1:56794)
0810 18:59:03.371865 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.371882 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:03.371987 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:03.372591 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:03.372593 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:03.372616 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:03.372705 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:03.372712 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.373792 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.373796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:03.375126 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:03.375157 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.376010 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.376014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:03.376115 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:03.376165 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 18:59:03.376252 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 18:59:03.376334 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"spinlock_wait_cycles":1792,"thread_start_us":56,"threads_started":1}
I20260810 18:59:03.376528 6972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.371635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.27.147.189:43471)
0810 18:59:03.372276 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:03.372280 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:03.372292 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:59:03.372333 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:03.372338 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:03.372384 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:03.372458 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:03.372831 (+ 373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.373669 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.375251 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:03.375889 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:03.376017 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:03.376158 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:03.376324 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 18:59:03.376402 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":48,"threads_started":1}
I20260810 18:59:03.377118 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c2b5bc1db18407da8e34c713db85f36" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f697e7c944a94e18bbfdb8c370670a63" is_pre_election: true
I20260810 18:59:03.377123 6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f697e7c944a94e18bbfdb8c370670a63" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c2b5bc1db18407da8e34c713db85f36" is_pre_election: true
I20260810 18:59:03.377306 5918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c2b5bc1db18407da8e34c713db85f36 in term 1.
I20260810 18:59:03.377305 6204 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f697e7c944a94e18bbfdb8c370670a63 in term 1.
I20260810 18:59:03.377669 5882 leader_election.cc:302] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c2b5bc1db18407da8e34c713db85f36, f697e7c944a94e18bbfdb8c370670a63; no voters: b007540de45341a0a6c5ef083c571502
I20260810 18:59:03.377673 6168 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c2b5bc1db18407da8e34c713db85f36, f697e7c944a94e18bbfdb8c370670a63; no voters: b007540de45341a0a6c5ef083c571502
I20260810 18:59:03.377866 6965 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:59:03.377926 6965 raft_consensus.cc:482] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b007540de45341a0a6c5ef083c571502)
I20260810 18:59:03.377966 6965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:59:03.378047 6966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 18:59:03.378099 6966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Starting leader election (detected failure of leader b007540de45341a0a6c5ef083c571502)
I20260810 18:59:03.378135 6966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:59:03.379108 6965 raft_consensus.cc:504] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:03.379176 6966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:03.379509 6965 leader_election.cc:288] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [CANDIDATE]: Term 2 election: Requested vote from peers b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793), 8c2b5bc1db18407da8e34c713db85f36 (127.27.147.188:32963)
I20260810 18:59:03.379580 6966 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 2 election: Requested vote from peers b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793), f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189:43471)
I20260810 18:59:03.379861 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.379574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34976)
0810 18:59:03.379749 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.379799 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20260810 18:59:03.379861 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.379657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34986)
0810 18:59:03.379750 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.379799 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260810 18:59:03.379957 6204 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f697e7c944a94e18bbfdb8c370670a63" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "8c2b5bc1db18407da8e34c713db85f36"
I20260810 18:59:03.380115 6204 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate f697e7c944a94e18bbfdb8c370670a63 in current term 2: Already voted for candidate 8c2b5bc1db18407da8e34c713db85f36 in this term.
I20260810 18:59:03.380095 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c2b5bc1db18407da8e34c713db85f36" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f697e7c944a94e18bbfdb8c370670a63"
I20260810 18:59:03.380240 5918 raft_consensus.cc:2379] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 8c2b5bc1db18407da8e34c713db85f36 in current term 2: Already voted for candidate f697e7c944a94e18bbfdb8c370670a63 in this term.
I20260810 18:59:03.380751 6974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.380292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34990)
0810 18:59:03.380656 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.380706 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302}
W20260810 18:59:03.380848 5881 leader_election.cc:334] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793): Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:03.380914 6971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.380722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:34998)
0810 18:59:03.380804 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.380857 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260810 18:59:03.380909 5881 leader_election.cc:302] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f697e7c944a94e18bbfdb8c370670a63; no voters: 8c2b5bc1db18407da8e34c713db85f36, b007540de45341a0a6c5ef083c571502
W20260810 18:59:03.381006 6169 leader_election.cc:334] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793): Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:03.381053 6965 raft_consensus.cc:2726] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:59:03.381104 6169 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c2b5bc1db18407da8e34c713db85f36; no voters: b007540de45341a0a6c5ef083c571502, f697e7c944a94e18bbfdb8c370670a63
I20260810 18:59:03.381247 6966 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260810 18:59:03.859370 6980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.858865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:46751)
0810 18:59:03.859173 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.859253 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":90,"threads_started":1}
I20260810 18:59:03.861754 6981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.861357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:53279)
0810 18:59:03.861582 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.861641 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":83,"threads_started":1}
I20260810 18:59:03.862033 6982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:03.861520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:42705)
0810 18:59:03.861841 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:03.861932 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":78,"threads_started":1}
I20260810 18:59:04.860502 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:04.860062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:59361)
0810 18:59:04.860342 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:04.860409 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":111,"threads_started":1}
I20260810 18:59:04.862937 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:04.862444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:33525)
0810 18:59:04.862719 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:04.862799 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":81,"threads_started":1}
I20260810 18:59:04.863593 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:04.862804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:49085)
0810 18:59:04.863429 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:04.863494 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":65,"threads_started":1}
I20260810 18:59:05.174430 6989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:05.174609 6989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:05.175153 6989 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793), f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189:43471)
I20260810 18:59:05.175630 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c2b5bc1db18407da8e34c713db85f36" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f697e7c944a94e18bbfdb8c370670a63" is_pre_election: true
I20260810 18:59:05.175814 5918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c2b5bc1db18407da8e34c713db85f36 in term 2.
I20260810 18:59:05.175841 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.175376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:35230)
0810 18:59:05.175599 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.175657 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":82,"threads_started":1}
I20260810 18:59:05.176201 6168 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8c2b5bc1db18407da8e34c713db85f36, f697e7c944a94e18bbfdb8c370670a63; no voters:
I20260810 18:59:05.176396 6989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260810 18:59:05.176501 6989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:05.176573 6989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:59:05.176784 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.176523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:35244)
0810 18:59:05.176675 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.176730 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 18:59:05.176894 6169 leader_election.cc:334] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793): Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:05.177731 6989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:05.178071 6989 leader_election.cc:288] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 3 election: Requested vote from peers b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793), f697e7c944a94e18bbfdb8c370670a63 (127.27.147.189:43471)
I20260810 18:59:05.178409 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.178224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:35252)
0810 18:59:05.178311 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.178360 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260810 18:59:05.178428 5918 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c2b5bc1db18407da8e34c713db85f36" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "f697e7c944a94e18bbfdb8c370670a63"
I20260810 18:59:05.178574 5918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 2 FOLLOWER]: Advancing to term 3
I20260810 18:59:05.179203 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.178948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:35260)
0810 18:59:05.179090 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.179146 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 18:59:05.179388 6169 leader_election.cc:334] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793): Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
I20260810 18:59:05.179705 5918 raft_consensus.cc:2454] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c2b5bc1db18407da8e34c713db85f36 in term 3.
I20260810 18:59:05.179981 6168 leader_election.cc:302] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8c2b5bc1db18407da8e34c713db85f36, f697e7c944a94e18bbfdb8c370670a63; no voters: b007540de45341a0a6c5ef083c571502
I20260810 18:59:05.180223 6989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 18:59:05.180500 6989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [term 3 LEADER]: Becoming Leader. State: Replica: 8c2b5bc1db18407da8e34c713db85f36, State: Running, Role: LEADER
I20260810 18:59:05.180794 6989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } }
I20260810 18:59:05.182178 6992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2b5bc1db18407da8e34c713db85f36. Latest consensus state: current_term: 3 leader_uuid: "8c2b5bc1db18407da8e34c713db85f36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:05.182303 6992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:05.182614 6994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:59:05.184877 6994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:59:05.185272 6994 catalog_manager.cc:1259] Loaded cluster ID: e659b0ff0a5c428caa87ff428685bbf3
I20260810 18:59:05.185343 6994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:59:05.186066 6994 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:59:05.186483 6994 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Loaded TSK: 0
I20260810 18:59:05.187227 6994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:59:05.195776 6177 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58796:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:59:05.199059 6177 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:05.201795 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.201560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:35282)
0810 18:59:05.201679 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.201738 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20260810 18:59:05.201911 5918 raft_consensus.cc:1261] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [term 3 FOLLOWER]: Refusing update from remote peer 8c2b5bc1db18407da8e34c713db85f36: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260810 18:59:05.202376 6992 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 18:59:05.202472 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.202239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.0.0.1:35292)
0810 18:59:05.202355 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.202413 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 18:59:05.202687 6169 consensus_peers.cc:487] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 -> Peer b007540de45341a0a6c5ef083c571502 (127.27.147.190:42793): Couldn't send request to peer b007540de45341a0a6c5ef083c571502. Status: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 18:59:05.203835 6997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c2b5bc1db18407da8e34c713db85f36. Latest consensus state: current_term: 3 leader_uuid: "8c2b5bc1db18407da8e34c713db85f36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:05.203950 6997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:59:05.205067 6992 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8c2b5bc1db18407da8e34c713db85f36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:05.205497 6998 mvcc.cc:205] Tried to move back new op lower bound from 7317046661942747136 to 7317046661863297024. Current Snapshot: MvccSnapshot[applied={T|T < 7317046661942747136}]
I20260810 18:59:05.205741 6997 sys_catalog.cc:455] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8c2b5bc1db18407da8e34c713db85f36" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "b007540de45341a0a6c5ef083c571502" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42793 } } peers { permanent_uuid: "f697e7c944a94e18bbfdb8c370670a63" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 43471 } attrs { promote: false } } peers { permanent_uuid: "8c2b5bc1db18407da8e34c713db85f36" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 32963 } attrs { promote: false } } }
I20260810 18:59:05.205850 6997 sys_catalog.cc:458] T 00000000000000000000000000000000 P f697e7c944a94e18bbfdb8c370670a63 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 18:59:05.205957 6992 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:05.215498 7006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.210830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37149 (local address 127.0.0.1:53762)
0810 18:59:05.211080 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.211096 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:05.211184 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:05.211689 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:05.211691 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:05.211708 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:05.211794 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.211798 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.213002 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.213005 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.214125 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.214134 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.215067 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.215070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.215161 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.215184 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 18:59:05.215285 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 18:59:05.215357 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":99,"threads_started":1}
I20260810 18:59:05.215498 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.211217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53762 (local address 127.27.147.129:37149)
0810 18:59:05.211458 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:05.211463 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:05.211477 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:05.211528 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:05.211531 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:05.211578 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:05.211665 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:05.211901 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.212879 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.214247 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.214961 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.215078 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.215133 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:05.215299 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 18:59:05.215372 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":164,"thread_start_us":65,"threads_started":1}
I20260810 18:59:05.216332 6351 tablet_service.cc:1467] Processing CreateTablet for tablet 28844bfc7b6241288b8964ad2aa4ab1e (DEFAULT_TABLE table=test-table [id=6eddad456d384dfaa0720c4f322ea9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:05.216653 6351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28844bfc7b6241288b8964ad2aa4ab1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:05.217097 6990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.210725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:38435 (local address 127.0.0.1:53310)
0810 18:59:05.210833 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.210847 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:05.210943 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:05.213567 (+ 2624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:05.213571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:05.213589 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:05.213667 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.213672 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.214555 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.214560 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.215810 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.215821 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.216691 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.216698 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.216794 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.216816 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:59:05.216892 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:59:05.216966 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 18:59:05.217257 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.212510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44075 (local address 127.0.0.1:59276)
0810 18:59:05.212721 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.212735 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:05.212819 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:05.213678 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:05.213681 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:05.213698 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:05.213789 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.213795 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.214704 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.214707 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.215870 (+ 1163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.215879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.216886 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.216890 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.216970 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.216991 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 18:59:05.217054 (+ 63us) client_negotiation.cc:240] Negotiation successful
0810 18:59:05.217122 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":4,"thread_start_us":87,"threads_started":1}
I20260810 18:59:05.217293 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.212229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.27.147.131:38435)
0810 18:59:05.213136 (+ 907us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:05.213142 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:05.213158 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:59:05.213386 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:05.213389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:05.213442 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:05.213536 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:05.213798 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.214431 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.215952 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.216510 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.216683 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.216734 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:05.216930 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 18:59:05.217035 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":822,"thread_start_us":93,"threads_started":1}
I20260810 18:59:05.217620 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.212625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59276 (local address 127.27.147.130:44075)
0810 18:59:05.213474 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:05.213480 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:05.213496 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:59:05.213545 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:05.213548 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:05.213588 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:05.213668 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:05.213910 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.214597 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.216000 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.216661 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.216776 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.216826 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:05.217394 (+ 568us) server_negotiation.cc:299] Negotiation successful
0810 18:59:05.217495 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":70,"threads_started":1}
I20260810 18:59:05.217783 6751 tablet_service.cc:1467] Processing CreateTablet for tablet 28844bfc7b6241288b8964ad2aa4ab1e (DEFAULT_TABLE table=test-table [id=6eddad456d384dfaa0720c4f322ea9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:05.218024 6751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28844bfc7b6241288b8964ad2aa4ab1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:05.219339 6547 tablet_service.cc:1467] Processing CreateTablet for tablet 28844bfc7b6241288b8964ad2aa4ab1e (DEFAULT_TABLE table=test-table [id=6eddad456d384dfaa0720c4f322ea9e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:05.219628 6547 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 28844bfc7b6241288b8964ad2aa4ab1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:05.220311 7015 tablet_bootstrap.cc:492] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: Bootstrap starting.
I20260810 18:59:05.221290 7015 tablet_bootstrap.cc:654] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:05.222086 7017 tablet_bootstrap.cc:492] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: Bootstrap starting.
I20260810 18:59:05.222191 7014 tablet_bootstrap.cc:492] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f: Bootstrap starting.
I20260810 18:59:05.222483 7015 tablet_bootstrap.cc:492] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: No bootstrap required, opened a new log
I20260810 18:59:05.222584 7015 ts_tablet_manager.cc:1397] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:59:05.223052 7017 tablet_bootstrap.cc:654] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:05.223032 7015 raft_consensus.cc:348] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.223115 7014 tablet_bootstrap.cc:654] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:05.223140 7015 raft_consensus.cc:374] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:05.223212 7015 raft_consensus.cc:729] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a533e704c5e4f4eaf7851abf9828d86, State: Initialized, Role: FOLLOWER
I20260810 18:59:05.223408 7015 consensus_queue.cc:260] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.223837 7015 ts_tablet_manager.cc:1428] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:59:05.224557 7017 tablet_bootstrap.cc:492] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: No bootstrap required, opened a new log
I20260810 18:59:05.224646 7017 ts_tablet_manager.cc:1397] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:05.224690 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.224405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:49739)
0810 18:59:05.224572 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.224625 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260810 18:59:05.225078 7017 raft_consensus.cc:348] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.225176 7017 raft_consensus.cc:374] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:05.225204 7017 raft_consensus.cc:729] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ab1c37242045b0806525c8ecbb76c3, State: Initialized, Role: FOLLOWER
I20260810 18:59:05.225335 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.225102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:46677)
0810 18:59:05.225231 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.225277 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 18:59:05.225309 7017 consensus_queue.cc:260] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.225675 7017 ts_tablet_manager.cc:1428] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:05.226727 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.226418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:53173)
0810 18:59:05.226586 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.226651 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 18:59:05.226915 6854 heartbeater.cc:503] Master 127.27.147.188:32963 was elected leader, sending a full tablet report...
I20260810 18:59:05.227381 7014 tablet_bootstrap.cc:492] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f: No bootstrap required, opened a new log
I20260810 18:59:05.227504 7014 ts_tablet_manager.cc:1397] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260810 18:59:05.228065 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.227713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:45099)
0810 18:59:05.227939 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.227998 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260810 18:59:05.228029 7014 raft_consensus.cc:348] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.228179 7014 raft_consensus.cc:374] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:05.228235 7014 raft_consensus.cc:729] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f34121eec10941dfbb6f650604c8656f, State: Initialized, Role: FOLLOWER
I20260810 18:59:05.228379 7014 consensus_queue.cc:260] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.228729 7014 ts_tablet_manager.cc:1428] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:05.229401 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.229082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:34311)
0810 18:59:05.229244 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.229309 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 18:59:05.230264 6438 heartbeater.cc:503] Master 127.27.147.188:32963 was elected leader, sending a full tablet report...
I20260810 18:59:05.230540 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.230264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:50599)
0810 18:59:05.230420 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.230472 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 18:59:05.284550 7026 raft_consensus.cc:482] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:05.284708 7026 raft_consensus.cc:504] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.285964 7026 leader_election.cc:288] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131:38435), f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
I20260810 18:59:05.289619 6983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.286177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37149 (local address 127.27.147.130:45665)
0810 18:59:05.286357 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.286372 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:05.286473 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:05.286735 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:05.286738 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:05.286758 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:05.286916 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.286922 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.287682 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.287688 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.288481 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.288490 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.289139 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.289143 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.289296 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.289324 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 18:59:05.289414 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 18:59:05.289498 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 18:59:05.289650 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.286312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:45665 (local address 127.27.147.129:37149)
0810 18:59:05.286447 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:05.286452 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:05.286479 (+ 27us) server_negotiation.cc:407] Connection header received
0810 18:59:05.286577 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:05.286580 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:05.286618 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:05.286720 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:05.287040 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.287563 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.288608 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.288999 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.289134 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.289192 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:05.289454 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 18:59:05.289549 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:59:05.289841 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.286175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:38435 (local address 127.27.147.130:45379)
0810 18:59:05.286576 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.286590 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:05.286715 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:05.287012 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:05.287015 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:05.287031 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:05.287150 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.287155 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.287873 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.287877 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.288628 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:05.288636 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.289324 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.289328 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:05.289457 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.289475 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 18:59:05.289603 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 18:59:05.289676 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":36,"thread_start_us":148,"threads_started":1}
I20260810 18:59:05.289829 7008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.286259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:45379 (local address 127.27.147.131:38435)
0810 18:59:05.286394 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:05.286401 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:05.286685 (+ 284us) server_negotiation.cc:407] Connection header received
0810 18:59:05.286860 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:05.286863 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:05.286912 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:05.286999 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:05.287262 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.287755 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.288755 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.289157 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.289329 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.289380 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:05.289633 (+ 253us) server_negotiation.cc:299] Negotiation successful
0810 18:59:05.289726 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:59:05.290445 6375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28844bfc7b6241288b8964ad2aa4ab1e" candidate_uuid: "c6ab1c37242045b0806525c8ecbb76c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34121eec10941dfbb6f650604c8656f" is_pre_election: true
I20260810 18:59:05.290511 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28844bfc7b6241288b8964ad2aa4ab1e" candidate_uuid: "c6ab1c37242045b0806525c8ecbb76c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a533e704c5e4f4eaf7851abf9828d86" is_pre_election: true
I20260810 18:59:05.290627 6375 raft_consensus.cc:2454] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ab1c37242045b0806525c8ecbb76c3 in term 0.
I20260810 18:59:05.290752 6779 raft_consensus.cc:2454] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ab1c37242045b0806525c8ecbb76c3 in term 0.
I20260810 18:59:05.291070 6482 leader_election.cc:302] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ab1c37242045b0806525c8ecbb76c3, f34121eec10941dfbb6f650604c8656f; no voters:
I20260810 18:59:05.291316 7026 raft_consensus.cc:2781] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:05.291376 7026 raft_consensus.cc:482] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:05.291415 7026 raft_consensus.cc:3037] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:05.292407 7026 raft_consensus.cc:504] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.292800 7026 leader_election.cc:288] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [CANDIDATE]: Term 1 election: Requested vote from peers 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131:38435), f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149)
I20260810 18:59:05.293102 6779 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28844bfc7b6241288b8964ad2aa4ab1e" candidate_uuid: "c6ab1c37242045b0806525c8ecbb76c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a533e704c5e4f4eaf7851abf9828d86"
I20260810 18:59:05.293102 6375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "28844bfc7b6241288b8964ad2aa4ab1e" candidate_uuid: "c6ab1c37242045b0806525c8ecbb76c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f34121eec10941dfbb6f650604c8656f"
I20260810 18:59:05.293218 6375 raft_consensus.cc:3037] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:05.293218 6779 raft_consensus.cc:3037] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:05.294397 6779 raft_consensus.cc:2454] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ab1c37242045b0806525c8ecbb76c3 in term 1.
I20260810 18:59:05.294397 6375 raft_consensus.cc:2454] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ab1c37242045b0806525c8ecbb76c3 in term 1.
I20260810 18:59:05.294690 6475 leader_election.cc:302] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a533e704c5e4f4eaf7851abf9828d86, c6ab1c37242045b0806525c8ecbb76c3; no voters:
I20260810 18:59:05.294938 7026 raft_consensus.cc:2781] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:05.295116 7026 raft_consensus.cc:686] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 1 LEADER]: Becoming Leader. State: Replica: c6ab1c37242045b0806525c8ecbb76c3, State: Running, Role: LEADER
I20260810 18:59:05.295411 6644 heartbeater.cc:503] Master 127.27.147.188:32963 was elected leader, sending a full tablet report...
I20260810 18:59:05.295372 7026 consensus_queue.cc:237] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } }
I20260810 18:59:05.295985 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.295575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:42295)
0810 18:59:05.295838 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.295891 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188}
I20260810 18:59:05.297516 6177 catalog_manager.cc:5495] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130). New cstate: current_term: 1 leader_uuid: "c6ab1c37242045b0806525c8ecbb76c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ab1c37242045b0806525c8ecbb76c3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44075 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 } health_report { overall_health: UNKNOWN } } }
I20260810 18:59:05.317202 7013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.314048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59286 (local address 127.27.147.130:44075)
0810 18:59:05.314174 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:05.314177 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:05.314189 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:59:05.314328 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:05.314331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:05.314375 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:05.314458 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:05.314662 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.315201 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.315981 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:05.316179 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:05.316252 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:05.316476 (+ 224us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:05.316866 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:05.316951 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:05.317006 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 18:59:05.317047 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:59:05.319590 6375 raft_consensus.cc:1261] T 28844bfc7b6241288b8964ad2aa4ab1e P f34121eec10941dfbb6f650604c8656f [term 1 FOLLOWER]: Refusing update from remote peer c6ab1c37242045b0806525c8ecbb76c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:05.319813 6779 raft_consensus.cc:1261] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 FOLLOWER]: Refusing update from remote peer c6ab1c37242045b0806525c8ecbb76c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:05.320094 7026 consensus_queue.cc:1035] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f34121eec10941dfbb6f650604c8656f" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37149 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:05.320370 7032 consensus_queue.cc:1035] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a533e704c5e4f4eaf7851abf9828d86" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:05.321557 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.321316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:34221)
0810 18:59:05.321432 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.321489 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 18:59:05.322628 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.322345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:37211)
0810 18:59:05.322508 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.322561 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 18:59:05.326324 6988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.326021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.129:38445)
0810 18:59:05.326207 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.326257 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260810 18:59:05.326885 6985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.326596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.131:60963)
0810 18:59:05.326744 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.326793 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 18:59:05.327579 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.327346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42793 (local address 127.27.147.130:44425)
0810 18:59:05.327464 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.327517 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:42793: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 18:59:05.334544 6177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:05.334679 6177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58796:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:05.338201 6177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c2b5bc1db18407da8e34c713db85f36: Sending DeleteTablet for 3 replicas of tablet 28844bfc7b6241288b8964ad2aa4ab1e
I20260810 18:59:05.338855 6547 tablet_service.cc:1514] Processing DeleteTablet for tablet 28844bfc7b6241288b8964ad2aa4ab1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:05 UTC) from {username='oryx', principal='oryx/127.27.147.188@KRBTEST.COM'} at 127.0.0.1:59276
I20260810 18:59:05.339197 7053 tablet_replica.cc:330] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: stopping tablet replica
I20260810 18:59:05.339330 7053 raft_consensus.cc:2229] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:59:05.339411 6751 tablet_service.cc:1514] Processing DeleteTablet for tablet 28844bfc7b6241288b8964ad2aa4ab1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:05 UTC) from {username='oryx', principal='oryx/127.27.147.188@KRBTEST.COM'} at 127.0.0.1:53310
I20260810 18:59:05.339692 7053 raft_consensus.cc:2258] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:05.339807 7054 tablet_replica.cc:330] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: stopping tablet replica
I20260810 18:59:05.339797 6351 tablet_service.cc:1514] Processing DeleteTablet for tablet 28844bfc7b6241288b8964ad2aa4ab1e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:05 UTC) from {username='oryx', principal='oryx/127.27.147.188@KRBTEST.COM'} at 127.0.0.1:53762
I20260810 18:59:05.340193 7054 raft_consensus.cc:2229] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:05.340373 7054 raft_consensus.cc:2258] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:05.340425 7053 ts_tablet_manager.cc:1905] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:05.340858 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 6121
I20260810 18:59:05.343986 7053 ts_tablet_manager.cc:1918] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:05.344132 7053 log.cc:1196] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/28844bfc7b6241288b8964ad2aa4ab1e
I20260810 18:59:05.344496 7053 ts_tablet_manager.cc:1939] T 28844bfc7b6241288b8964ad2aa4ab1e P c6ab1c37242045b0806525c8ecbb76c3: Deleting consensus metadata
I20260810 18:59:05.345279 6168 catalog_manager.cc:4841] TS c6ab1c37242045b0806525c8ecbb76c3 (127.27.147.130:44075): tablet 28844bfc7b6241288b8964ad2aa4ab1e (table test-table [id=6eddad456d384dfaa0720c4f322ea9e3]) successfully deleted
I20260810 18:59:05.347654 7054 ts_tablet_manager.cc:1905] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:05.350850 7054 ts_tablet_manager.cc:1918] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:05.351356 7054 log.cc:1196] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/28844bfc7b6241288b8964ad2aa4ab1e
I20260810 18:59:05.351694 7054 ts_tablet_manager.cc:1939] T 28844bfc7b6241288b8964ad2aa4ab1e P 2a533e704c5e4f4eaf7851abf9828d86: Deleting consensus metadata
I20260810 18:59:05.355020 6166 catalog_manager.cc:4841] TS 2a533e704c5e4f4eaf7851abf9828d86 (127.27.147.131:38435): tablet 28844bfc7b6241288b8964ad2aa4ab1e (table test-table [id=6eddad456d384dfaa0720c4f322ea9e3]) successfully deleted
I20260810 18:59:05.358526 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 6447
I20260810 18:59:05.358603 7012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:05.358309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37149 (local address 127.0.0.1:53764)
0810 18:59:05.358458 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:05.358517 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:37149: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260810 18:59:05.365478 6169 catalog_manager.cc:4584] TS f34121eec10941dfbb6f650604c8656f (127.27.147.129:37149): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 28844bfc7b6241288b8964ad2aa4ab1e: Network error: Client connection negotiation failed: client connection to 127.27.147.129:37149: connect: Connection refused (error 111)
I20260810 18:59:05.373051 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 6655
I20260810 18:59:05.384069 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 5703
I20260810 18:59:05.393123 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 5862
2026-08-10T18:59:05Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7548 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:59:05 dist-test-slave-znh6 krb5kdc[7061](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:59:05 dist-test-slave-znh6 krb5kdc[7061](info): set up 2 sockets
Aug 10 18:59:05 dist-test-slave-znh6 krb5kdc[7061](info): commencing operation
krb5kdc: starting...
W20260810 18:59:06.660549 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:59:06 dist-test-slave-znh6 krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:59:06Z Disabled control of system clock
WARNING: no policy specified for oryx/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab.
Entry for principal oryx/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab.
I20260810 18:59:06.781733 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38065
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab
--principal=oryx/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38065
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:06.892267 7083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:06.892526 7083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:06.892609 7083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:06.896606 7083 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:06.896683 7083 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 18:59:06.896718 7083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:06.896778 7083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:06.896824 7083 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:06.896857 7083 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:06.901887 7083 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38065
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.190.keytab
--principal=oryx/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7083
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:06.902909 7083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:06.903895 7083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:06.912206 7089 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:06.912206 7093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:06.912230 7090 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:06.912240 7083 server_base.cc:1034] running on GCE node
I20260810 18:59:06.913182 7083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:06.914270 7083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:06.915463 7083 hybrid_clock.cc:648] HybridClock initialized: now 1786388346915440 us; error 36 us; skew 500 ppm
Aug 10 18:59:06 dist-test-slave-znh6 krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388346, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:06.918279 7083 init.cc:377] Logged in from keytab as oryx/127.27.147.190@KRBTEST.COM (short username oryx)
I20260810 18:59:06.919201 7083 webserver.cc:460] Webserver started at http://127.27.147.190:46241/ using document root <none> and password file <none>
I20260810 18:59:06.919699 7083 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:06.919757 7083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:06.919926 7083 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:06.921859 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "b108f3b0aac34bdcabbdb5d67165c880"
format_stamp: "Formatted at 2026-08-10 18:59:06 on dist-test-slave-znh6"
server_key: "cae78a321cf43a0d8f5af9d7ca322d12"
server_key_iv: "6133d8affe9de9f4284514821d6cbabb"
server_key_version: "encryptionkey@0"
I20260810 18:59:06.922369 7083 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "b108f3b0aac34bdcabbdb5d67165c880"
format_stamp: "Formatted at 2026-08-10 18:59:06 on dist-test-slave-znh6"
server_key: "cae78a321cf43a0d8f5af9d7ca322d12"
server_key_iv: "6133d8affe9de9f4284514821d6cbabb"
server_key_version: "encryptionkey@0"
I20260810 18:59:06.925971 7083 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:59:06.928375 7102 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:06.929600 7083 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:59:06.929718 7083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "b108f3b0aac34bdcabbdb5d67165c880"
format_stamp: "Formatted at 2026-08-10 18:59:06 on dist-test-slave-znh6"
server_key: "cae78a321cf43a0d8f5af9d7ca322d12"
server_key_iv: "6133d8affe9de9f4284514821d6cbabb"
server_key_version: "encryptionkey@0"
I20260810 18:59:06.929801 7083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:06.948743 7083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:06.951848 7083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:06.951994 7083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:06.960270 7083 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38065
I20260810 18:59:06.960289 7189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38065 every 8 connection(s)
I20260810 18:59:06.961193 7083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:59:06.964035 7190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:06.967942 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 7083
I20260810 18:59:06.968077 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:59:06.968379 28238 external_mini_cluster.cc:1428] Setting key e0cda01836de1027a570d3fde0180738
I20260810 18:59:06.970069 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Bootstrap starting.
I20260810 18:59:06.972592 7190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:06.973323 7190 log.cc:824] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:06.975512 7190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: No bootstrap required, opened a new log
Aug 10 18:59:06 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388346, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:06.980058 7190 raft_consensus.cc:348] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b108f3b0aac34bdcabbdb5d67165c880" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38065 } }
I20260810 18:59:06.980326 7190 raft_consensus.cc:374] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:06.980415 7190 raft_consensus.cc:729] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b108f3b0aac34bdcabbdb5d67165c880, State: Initialized, Role: FOLLOWER
I20260810 18:59:06.980906 7190 consensus_queue.cc:260] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b108f3b0aac34bdcabbdb5d67165c880" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38065 } }
I20260810 18:59:06.981048 7190 raft_consensus.cc:388] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:59:06.981135 7190 raft_consensus.cc:482] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:59:06.981251 7190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:06.982136 7190 raft_consensus.cc:504] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b108f3b0aac34bdcabbdb5d67165c880" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38065 } }
I20260810 18:59:06.982489 7190 leader_election.cc:302] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b108f3b0aac34bdcabbdb5d67165c880; no voters:
I20260810 18:59:06.982790 7190 leader_election.cc:288] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:59:06.982966 7198 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:06.983177 7198 raft_consensus.cc:686] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [term 1 LEADER]: Becoming Leader. State: Replica: b108f3b0aac34bdcabbdb5d67165c880, State: Running, Role: LEADER
I20260810 18:59:06.983266 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:06.969752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59422 (local address 127.27.147.190:38065)
0810 18:59:06.970154 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:06.970168 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:06.970231 (+ 63us) server_negotiation.cc:407] Connection header received
0810 18:59:06.970896 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:06.970913 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:06.971219 (+ 306us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:06.971484 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:06.972367 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:06.973454 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:06.974505 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:06.975023 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:06.975144 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:06.977963 (+ 2819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:06.977970 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:06.977986 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:06.978017 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:06.979954 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:06.980402 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:06.980407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:06.980412 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:06.980577 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:06.980882 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:06.980885 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:06.980887 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:06.981455 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:06.981666 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:06.981963 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 18:59:06.982279 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":128,"threads_started":1}
I20260810 18:59:06.983896 7198 consensus_queue.cc:237] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b108f3b0aac34bdcabbdb5d67165c880" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38065 } }
I20260810 18:59:06.984120 7190 sys_catalog.cc:564] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:06.985585 7200 sys_catalog.cc:455] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b108f3b0aac34bdcabbdb5d67165c880. Latest consensus state: current_term: 1 leader_uuid: "b108f3b0aac34bdcabbdb5d67165c880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b108f3b0aac34bdcabbdb5d67165c880" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38065 } } }
I20260810 18:59:06.985841 7201 sys_catalog.cc:455] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b108f3b0aac34bdcabbdb5d67165c880" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b108f3b0aac34bdcabbdb5d67165c880" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38065 } } }
I20260810 18:59:06.985971 7200 sys_catalog.cc:458] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:06.986029 7201 sys_catalog.cc:458] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:06.986452 7207 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:59:06.989585 7207 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:59:06.994954 7207 catalog_manager.cc:1347] Generated new cluster ID: c7cb2728eb4f48c59c67ceccaed4f10a
I20260810 18:59:06.995019 7207 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:59:07.006289 7207 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:59:07.007179 7207 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:59:07.017536 7207 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Generated new TSK 0
I20260810 18:59:07.018272 7207 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab.
Entry for principal oryx/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab.
I20260810 18:59:07.062328 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38065
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab
--principal=oryx/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:07.203042 7230 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:07.203311 7230 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:07.203361 7230 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:07.207304 7230 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:07.207357 7230 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:07.207394 7230 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:59:07.207456 7230 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:59:07.212426 7230 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab
--principal=oryx/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:38065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7230
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:07.213367 7230 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:07.214309 7230 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:07.223706 7243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:07.223727 7230 server_base.cc:1034] running on GCE node
W20260810 18:59:07.223672 7239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:07.223664 7240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:07.224431 7230 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:07.225065 7230 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:07.226248 7230 hybrid_clock.cc:648] HybridClock initialized: now 1786388347226219 us; error 39 us; skew 500 ppm
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:07.229251 7230 init.cc:377] Logged in from keytab as oryx/127.27.147.129@KRBTEST.COM (short username oryx)
I20260810 18:59:07.230027 7230 webserver.cc:460] Webserver started at http://127.27.147.129:35911/ using document root <none> and password file <none>
I20260810 18:59:07.230540 7230 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:07.230616 7230 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:07.230777 7230 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:07.232859 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "aee157df2c2045158ead0ec9b2f7ea7e"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "13e9a8c4948dc87ecc5153791af86053"
server_key_iv: "78822de61c1fdc541f286b071b6d4149"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.233497 7230 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "aee157df2c2045158ead0ec9b2f7ea7e"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "13e9a8c4948dc87ecc5153791af86053"
server_key_iv: "78822de61c1fdc541f286b071b6d4149"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.237215 7230 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:59:07.239513 7251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:07.240768 7230 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:07.240883 7230 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "aee157df2c2045158ead0ec9b2f7ea7e"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "13e9a8c4948dc87ecc5153791af86053"
server_key_iv: "78822de61c1fdc541f286b071b6d4149"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.240984 7230 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:07.255224 7230 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:07.258356 7230 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:07.258500 7230 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:07.260249 7230 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:07.260298 7230 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:07.260373 7230 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:07.260407 7230 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:07.271121 7230 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:33651
I20260810 18:59:07.271211 7422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:33651 every 8 connection(s)
I20260810 18:59:07.272142 7230 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:59:07.274780 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.261088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:45273 (local address 127.27.147.190:38065)
0810 18:59:07.261269 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.261277 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.262044 (+ 767us) server_negotiation.cc:407] Connection header received
0810 18:59:07.263280 (+ 1236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.263284 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.263353 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.263460 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.264973 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.265888 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.267518 (+ 1630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.267989 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.268109 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.270757 (+ 2648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:07.270761 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:07.270764 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:07.270798 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:07.272376 (+ 1578us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.272961 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.272966 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.272970 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.273062 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.273411 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.273535 (+ 124us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.273539 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.273804 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:07.273923 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.274401 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.274547 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:59:07.275475 7261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.261416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38065 (local address 127.27.147.129:45273)
0810 18:59:07.261869 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.261916 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.262978 (+ 1062us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.263668 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.263679 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.264136 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:07.264784 (+ 648us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.264803 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.266029 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.266033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.267405 (+ 1372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.267415 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.268194 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.268198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.268380 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.269021 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:07.269047 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:07.270589 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:07.272493 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.272499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.272512 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.272866 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.273156 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.273159 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.273161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.273300 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.273946 (+ 646us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:07.273951 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:07.274164 (+ 213us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.274325 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.274599 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":139,"threads_started":1}
I20260810 18:59:07.276737 7425 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38065
I20260810 18:59:07.276882 7425 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:07.277448 7425 heartbeater.cc:511] Master 127.27.147.190:38065 requested a full tablet report, sending...
I20260810 18:59:07.279086 7123 ts_manager.cc:194] Registered new tserver with Master: aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651)
I20260810 18:59:07.279659 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 7230
I20260810 18:59:07.279790 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:59:07.280122 28238 external_mini_cluster.cc:1428] Setting key 39c382eebea7e254e67b795330d24a79
I20260810 18:59:07.280591 7123 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:45273
I20260810 18:59:07.289489 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.278999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:34235 (local address 127.27.147.190:38065)
0810 18:59:07.279113 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.279117 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.281381 (+ 2264us) server_negotiation.cc:407] Connection header received
0810 18:59:07.281603 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.281606 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.281655 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.281746 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.282233 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.283058 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.284069 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.284279 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.284357 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.285788 (+ 1431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:07.285791 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:07.285793 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:07.285815 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:07.287804 (+ 1989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.288316 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.288319 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.288320 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.288393 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.288713 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.288715 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.288716 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.288869 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:07.288946 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.289280 (+ 334us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.289370 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 18:59:07.289681 7435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.278922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38065 (local address 127.27.147.129:34235)
0810 18:59:07.281295 (+ 2373us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.281309 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.281435 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.281780 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.281783 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.281988 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:07.282097 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.282104 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.283184 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.283187 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.283940 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.283954 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.284399 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.284404 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.284524 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.285275 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:07.285295 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:07.285643 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:07.287931 (+ 2288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.287937 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.287942 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.288214 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.288491 (+ 277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.288497 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.288502 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.288612 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.288968 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:07.288975 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:07.289094 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.289223 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.289389 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2286,"thread_start_us":93,"threads_started":1}
WARNING: no policy specified for oryx/127.27.147.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab.
Entry for principal oryx/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab.
I20260810 18:59:07.309607 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38065
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab
--principal=oryx/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:07.422866 7441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:07.423125 7441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:07.423198 7441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:07.427256 7441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:07.427309 7441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:07.427348 7441 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:59:07.427424 7441 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:59:07.432775 7441 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab
--principal=oryx/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:38065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:07.433868 7441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:07.434939 7441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:07.444833 7451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:07.444859 7448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:07.444833 7447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:07.445098 7441 server_base.cc:1034] running on GCE node
I20260810 18:59:07.445677 7441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:07.446291 7441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:07.447476 7441 hybrid_clock.cc:648] HybridClock initialized: now 1786388347447459 us; error 25 us; skew 500 ppm
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:07.450114 7441 init.cc:377] Logged in from keytab as oryx/127.27.147.130@KRBTEST.COM (short username oryx)
I20260810 18:59:07.451059 7441 webserver.cc:460] Webserver started at http://127.27.147.130:39739/ using document root <none> and password file <none>
I20260810 18:59:07.451625 7441 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:07.451733 7441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:07.451973 7441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:07.453969 7441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "09050963869d44b6a9e7268448139048"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "e9b8ff63b7bafe75b6fa3af2e95cbbbf"
server_key_iv: "ff40d9c71889e10b3825051f3cbaa9ca"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.454566 7441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "09050963869d44b6a9e7268448139048"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "e9b8ff63b7bafe75b6fa3af2e95cbbbf"
server_key_iv: "ff40d9c71889e10b3825051f3cbaa9ca"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.458243 7441 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:07.460711 7461 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:07.461889 7441 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:59:07.462028 7441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "09050963869d44b6a9e7268448139048"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "e9b8ff63b7bafe75b6fa3af2e95cbbbf"
server_key_iv: "ff40d9c71889e10b3825051f3cbaa9ca"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.462137 7441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:07.483316 7441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:07.486272 7441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:07.486478 7441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:07.488226 7441 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:07.488301 7441 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:07.488374 7441 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:07.488415 7441 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:07.500026 7441 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:43639
I20260810 18:59:07.500111 7635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:43639 every 8 connection(s)
I20260810 18:59:07.500914 7441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:59:07.502529 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.488899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:53237 (local address 127.27.147.190:38065)
0810 18:59:07.489052 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.489057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.489961 (+ 904us) server_negotiation.cc:407] Connection header received
0810 18:59:07.491174 (+ 1213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.491178 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.491237 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.491320 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.492647 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.493384 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.495047 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.495435 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.495544 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.498701 (+ 3157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:07.498705 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:07.498708 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:07.498736 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:07.500392 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.500920 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.500925 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.500927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.501010 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.501349 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.501354 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.501359 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.501606 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:07.501734 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.502223 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.502358 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:59:07.503378 7477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.489321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38065 (local address 127.27.147.130:53237)
0810 18:59:07.489774 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.489828 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.490960 (+ 1132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.491592 (+ 632us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.491599 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.491907 (+ 308us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:07.492498 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.492515 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.493537 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.493540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.494895 (+ 1355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.494908 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.495579 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.495583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.495817 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.496483 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:07.496508 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:07.498538 (+ 2030us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:07.500510 (+ 1972us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.500515 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.500529 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.500812 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.501105 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.501109 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.501111 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.501249 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.501759 (+ 510us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:07.501765 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:07.501990 (+ 225us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.502180 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.502498 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":127,"threads_started":1}
I20260810 18:59:07.504709 7637 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38065
I20260810 18:59:07.504892 7637 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:07.505422 7637 heartbeater.cc:511] Master 127.27.147.190:38065 requested a full tablet report, sending...
I20260810 18:59:07.506390 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 7441
I20260810 18:59:07.506526 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:59:07.506594 7125 ts_manager.cc:194] Registered new tserver with Master: 09050963869d44b6a9e7268448139048 (127.27.147.130:43639)
I20260810 18:59:07.506877 28238 external_mini_cluster.cc:1428] Setting key c392d5499d90d45f9cd010d8c3769195
I20260810 18:59:07.507452 7125 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:53237
I20260810 18:59:07.516076 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.507308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:54283 (local address 127.27.147.190:38065)
0810 18:59:07.507404 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.507408 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.507422 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:07.507480 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.507484 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.507530 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.507608 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.509413 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.510196 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.511543 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.511750 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.511833 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.513013 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:07.513016 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:07.513019 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:07.513043 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:07.514429 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.514867 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.514872 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.514876 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.514962 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.515239 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.515243 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.515245 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.515452 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:07.515567 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.515829 (+ 262us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.515938 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:59:07.516091 7645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.506991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38065 (local address 127.27.147.130:54283)
0810 18:59:07.507285 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.507301 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.507397 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.507628 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.507630 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.509207 (+ 1577us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:07.509294 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.509300 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.510371 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.510377 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.511417 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.511437 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.511868 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.511873 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.512008 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.512569 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:07.512580 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:07.512868 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:07.514551 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.514555 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.514557 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.514752 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.515055 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.515060 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.515062 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.515146 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.515560 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:07.515563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:07.515661 (+ 98us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.515764 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.515901 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":97,"threads_started":1}
WARNING: no policy specified for oryx/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab.
Entry for principal oryx/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab.
I20260810 18:59:07.538914 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38065
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab
--principal=oryx/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40917
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:07.654291 7648 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:07.654538 7648 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:07.654582 7648 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:07.658529 7648 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:07.658586 7648 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:07.658655 7648 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 18:59:07.658736 7648 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:07.663909 7648 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40917
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab
--principal=oryx/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.27.147.190:38065
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7648
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:07.664912 7648 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:07.665992 7648 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:07.675649 7661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:07.675658 7656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:07.675806 7648 server_base.cc:1034] running on GCE node
W20260810 18:59:07.675664 7657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:07.676302 7648 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:07.676975 7648 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:07.678148 7648 hybrid_clock.cc:648] HybridClock initialized: now 1786388347678115 us; error 43 us; skew 500 ppm
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:07.680830 7648 init.cc:377] Logged in from keytab as oryx/127.27.147.131@KRBTEST.COM (short username oryx)
I20260810 18:59:07.681681 7648 webserver.cc:460] Webserver started at http://127.27.147.131:42459/ using document root <none> and password file <none>
I20260810 18:59:07.682232 7648 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:07.682323 7648 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:07.682548 7648 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:07.684628 7648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "b0e22c4a7f264aba99136dfb225630d6"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "7ed38014131a75fcaac2514dd08e8084"
server_key_iv: "a933916582ded23bfdefd62d5e29e1ee"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.685271 7648 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "b0e22c4a7f264aba99136dfb225630d6"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "7ed38014131a75fcaac2514dd08e8084"
server_key_iv: "a933916582ded23bfdefd62d5e29e1ee"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.689126 7648 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 18:59:07.691465 7669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:07.692591 7648 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:07.692729 7648 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "b0e22c4a7f264aba99136dfb225630d6"
format_stamp: "Formatted at 2026-08-10 18:59:07 on dist-test-slave-znh6"
server_key: "7ed38014131a75fcaac2514dd08e8084"
server_key_iv: "a933916582ded23bfdefd62d5e29e1ee"
server_key_version: "encryptionkey@0"
I20260810 18:59:07.692850 7648 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:07.722697 7648 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:07.725543 7648 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:07.725754 7648 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:07.727293 7648 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:07.727368 7648 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:07.727463 7648 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:07.727512 7648 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:07.738896 7648 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:38751
I20260810 18:59:07.739161 7833 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:38751 every 8 connection(s)
I20260810 18:59:07.739936 7648 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:07.741650 7681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.728309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38065 (local address 127.27.147.131:39493)
0810 18:59:07.728732 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.728779 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.729623 (+ 844us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.730185 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.730194 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.730677 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:07.731210 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.731223 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.732080 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.732083 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.732976 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.732985 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.733517 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.733525 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.733754 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.734349 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:07.734387 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:07.737118 (+ 2731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:07.738967 (+ 1849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.738972 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.738986 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.739339 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.739646 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.739648 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.739649 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.739770 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.740214 (+ 444us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:07.740218 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:07.740476 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.740677 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.740971 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":125,"threads_started":1}
I20260810 18:59:07.742209 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.727893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:39493 (local address 127.27.147.190:38065)
0810 18:59:07.728043 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.728048 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.728906 (+ 858us) server_negotiation.cc:407] Connection header received
0810 18:59:07.729830 (+ 924us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.729835 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.729904 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.730005 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.731369 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.731948 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.733108 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.733357 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.733492 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.737281 (+ 3789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:07.737285 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:07.737288 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:07.737318 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:07.738797 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.739435 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.739439 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.739441 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.739546 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.739866 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.739870 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.739874 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.740112 (+ 238us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:07.740225 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.742005 (+ 1780us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.742104 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 18:59:07.743222 7838 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38065
I20260810 18:59:07.743415 7838 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:07.743974 7838 heartbeater.cc:511] Master 127.27.147.190:38065 requested a full tablet report, sending...
I20260810 18:59:07.745107 7125 ts_manager.cc:194] Registered new tserver with Master: b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751)
I20260810 18:59:07.745927 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 7648
I20260810 18:59:07.745913 7125 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:39493
I20260810 18:59:07.746052 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:59:07.746399 28238 external_mini_cluster.cc:1428] Setting key 54f9aa3e39305fd680e87b67faa4aaae
I20260810 18:59:07.748593 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:59:07.753126 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.745608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:40987 (local address 127.27.147.190:38065)
0810 18:59:07.745763 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.745769 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.745783 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:07.745814 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.745816 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.745857 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.745937 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.746426 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.747184 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.748540 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.748833 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.748931 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.749956 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:07.749960 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:07.749963 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:07.749996 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:07.751532 (+ 1536us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.751961 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.751967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.751971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.752050 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.752335 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.752339 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.752340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.752512 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:07.752618 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.752868 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.752976 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:59:07.753145 7847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.745276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38065 (local address 127.27.147.131:40987)
0810 18:59:07.745583 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.745603 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.745702 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.745953 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.745963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.746187 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:07.746308 (+ 121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.746316 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.747515 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.747518 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.748406 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.748417 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.748933 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.748936 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.749008 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.749491 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:07.749512 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:07.749813 (+ 301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:07.751644 (+ 1831us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.751649 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.751651 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.751845 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.752144 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:07.752148 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:07.752150 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:07.752234 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:07.752609 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:07.752612 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:07.752708 (+ 96us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.752815 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.752951 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:59:07.774502 7195 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:07.768426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59430 (local address 127.27.147.190:38065)
0810 18:59:07.768565 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.768568 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.768579 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:59:07.768612 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.768614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.768647 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.768736 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.769452 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.769959 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.770840 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.771059 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.771167 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.774336 (+ 3169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59430: BlockingRecv error: recv got EOF from 127.0.0.1:59430 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 18:59:07 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388347, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:07.783695 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.775927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59442 (local address 127.27.147.190:38065)
0810 18:59:07.776061 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.776067 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.776253 (+ 186us) server_negotiation.cc:407] Connection header received
0810 18:59:07.776325 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.776327 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.776363 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.776429 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:07.777068 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.777621 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.778428 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.778630 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.778711 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.780847 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:07.780850 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:07.780852 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:07.780873 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:07.782090 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.782483 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.782486 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.782487 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.782564 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:07.782798 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:07.782800 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:07.782801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:07.783055 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:07.783159 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.783395 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.783524 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:59:07.786692 7125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:59:07.789780 7125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:07.805075 7876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.798799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:33651 (local address 127.0.0.1:51230)
0810 18:59:07.799868 (+ 1069us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.799922 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.800050 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.800635 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.800642 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.800664 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:07.800790 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.800796 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.802259 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.802261 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.803558 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.803569 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.804497 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.804501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.804616 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.804670 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.804784 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.804905 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":913,"mutex_wait_us":37,"spinlock_wait_cycles":15744,"thread_start_us":97,"threads_started":1}
I20260810 18:59:07.805085 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.799556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51230 (local address 127.27.147.129:33651)
0810 18:59:07.800191 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.800198 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.800216 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:59:07.800293 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.800300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.800478 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.800631 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:07.800922 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.802142 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.803704 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.804377 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.804478 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.804563 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.804832 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.804953 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":73,"threads_started":1}
I20260810 18:59:07.805140 7873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.798714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:38751 (local address 127.0.0.1:41146)
0810 18:59:07.799868 (+ 1154us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.799923 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.800066 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.800766 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.800769 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.800786 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:07.800891 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.800897 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.802252 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.802255 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.803686 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.803695 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.804732 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.804735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.804841 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.804865 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.804947 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.805013 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":235,"thread_start_us":70,"threads_started":1}
I20260810 18:59:07.805583 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.798839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.27.147.131:38751)
0810 18:59:07.799275 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.799292 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.800009 (+ 717us) server_negotiation.cc:407] Connection header received
0810 18:59:07.800221 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.800227 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.800449 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.800577 (+ 128us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:07.801009 (+ 432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.802107 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.803812 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.804608 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.804751 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.804885 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.805315 (+ 430us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.805420 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":97,"threads_started":1}
I20260810 18:59:07.805699 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.799636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.27.147.130:43639)
0810 18:59:07.799942 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:07.799954 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:07.800183 (+ 229us) server_negotiation.cc:407] Connection header received
0810 18:59:07.800419 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:07.800427 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:07.800588 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:07.800726 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:07.801107 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.801864 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.803348 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.804022 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.804210 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.804354 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:07.805428 (+ 1074us) server_negotiation.cc:299] Negotiation successful
0810 18:59:07.805530 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209,"thread_start_us":104,"threads_started":1}
I20260810 18:59:07.805337 7872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:07.799457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43639 (local address 127.0.0.1:55122)
0810 18:59:07.800082 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:07.800096 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:07.800260 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:07.800877 (+ 617us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:07.800880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:07.800894 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:07.800982 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.800991 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.801986 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.801990 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.803198 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:07.803208 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:07.804948 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:07.804951 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:07.805033 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:07.805055 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 18:59:07.805130 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 18:59:07.805214 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":87,"threads_started":1}
I20260810 18:59:07.808274 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 19d3f66226c341f9bea384d12e70695d (DEFAULT_TABLE table=test-table [id=eb8e3feccb4f4673a293d2030b688677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:07.808624 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 19d3f66226c341f9bea384d12e70695d (DEFAULT_TABLE table=test-table [id=eb8e3feccb4f4673a293d2030b688677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:07.808621 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 19d3f66226c341f9bea384d12e70695d (DEFAULT_TABLE table=test-table [id=eb8e3feccb4f4673a293d2030b688677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:07.809324 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d3f66226c341f9bea384d12e70695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:07.809636 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d3f66226c341f9bea384d12e70695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:07.809633 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19d3f66226c341f9bea384d12e70695d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:07.815301 7882 tablet_bootstrap.cc:492] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Bootstrap starting.
I20260810 18:59:07.817135 7883 tablet_bootstrap.cc:492] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Bootstrap starting.
I20260810 18:59:07.817582 7882 tablet_bootstrap.cc:654] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:07.818464 7882 log.cc:824] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:07.819111 7885 tablet_bootstrap.cc:492] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Bootstrap starting.
I20260810 18:59:07.819279 7883 tablet_bootstrap.cc:654] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:07.820079 7883 log.cc:824] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:07.820556 7882 tablet_bootstrap.cc:492] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: No bootstrap required, opened a new log
I20260810 18:59:07.820767 7882 ts_tablet_manager.cc:1397] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:59:07.821514 7885 tablet_bootstrap.cc:654] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:07.822153 7883 tablet_bootstrap.cc:492] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: No bootstrap required, opened a new log
I20260810 18:59:07.822250 7885 log.cc:824] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:07.822356 7883 ts_tablet_manager.cc:1397] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:59:07.824245 7885 tablet_bootstrap.cc:492] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: No bootstrap required, opened a new log
I20260810 18:59:07.824441 7885 ts_tablet_manager.cc:1397] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:59:07.824689 7882 raft_consensus.cc:348] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:07.824919 7882 raft_consensus.cc:374] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:07.824985 7882 raft_consensus.cc:729] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee157df2c2045158ead0ec9b2f7ea7e, State: Initialized, Role: FOLLOWER
I20260810 18:59:07.825413 7882 consensus_queue.cc:260] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:07.826033 7425 heartbeater.cc:503] Master 127.27.147.190:38065 was elected leader, sending a full tablet report...
I20260810 18:59:07.826152 7883 raft_consensus.cc:348] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:07.826340 7882 ts_tablet_manager.cc:1428] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Time spent starting tablet: real 0.005s user 0.001s sys 0.005s
I20260810 18:59:07.826380 7883 raft_consensus.cc:374] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:07.826448 7883 raft_consensus.cc:729] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09050963869d44b6a9e7268448139048, State: Initialized, Role: FOLLOWER
I20260810 18:59:07.826900 7883 consensus_queue.cc:260] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:07.827812 7883 ts_tablet_manager.cc:1428] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 18:59:07.828142 7637 heartbeater.cc:503] Master 127.27.147.190:38065 was elected leader, sending a full tablet report...
I20260810 18:59:07.829413 7885 raft_consensus.cc:348] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:07.829672 7885 raft_consensus.cc:374] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:07.829722 7885 raft_consensus.cc:729] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e22c4a7f264aba99136dfb225630d6, State: Initialized, Role: FOLLOWER
I20260810 18:59:07.830156 7885 consensus_queue.cc:260] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:07.830866 7838 heartbeater.cc:503] Master 127.27.147.190:38065 was elected leader, sending a full tablet report...
I20260810 18:59:07.831061 7885 ts_tablet_manager.cc:1428] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
W20260810 18:59:07.991175 7841 tablet.cc:2369] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:59:08.002293 7638 tablet.cc:2369] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 18:59:08.023556 7428 tablet.cc:2369] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:59:08.060434 7891 raft_consensus.cc:482] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:08.060658 7891 raft_consensus.cc:504] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:08.061947 7891 leader_election.cc:288] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751), 09050963869d44b6a9e7268448139048 (127.27.147.130:43639)
I20260810 18:59:08.067201 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.062408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:51145 (local address 127.27.147.131:38751)
0810 18:59:08.062569 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.062576 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.062593 (+ 17us) server_negotiation.cc:407] Connection header received
0810 18:59:08.062832 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.062837 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.062898 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.062981 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:08.063316 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.064151 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.065815 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.066484 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.066613 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.066668 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.066984 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.067062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 18:59:08.067201 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.062761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:39755 (local address 127.27.147.130:43639)
0810 18:59:08.062909 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.062915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.063116 (+ 201us) server_negotiation.cc:407] Connection header received
0810 18:59:08.063221 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.063223 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.063293 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.063371 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:08.063613 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.064322 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.065808 (+ 1486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.066484 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.066613 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.066668 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.066984 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.067062 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:59:08.067291 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.062647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43639 (local address 127.27.147.129:39755)
0810 18:59:08.063038 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.063050 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.063138 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.063388 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.063390 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.063405 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:08.063497 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.063501 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.064410 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.064413 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.065650 (+ 1237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.065656 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.066614 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.066618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.066828 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.066873 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.066953 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.067097 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":333,"mutex_wait_us":153,"thread_start_us":124,"threads_started":1}
I20260810 18:59:08.067473 7894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.062158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:38751 (local address 127.27.147.129:51145)
0810 18:59:08.062505 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.062523 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.062660 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.063011 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.063014 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.063043 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:08.063190 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.063199 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.064258 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.064261 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.065641 (+ 1380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.065652 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.066614 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.066618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.066781 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.066871 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.066953 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.067097 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":99,"thread_start_us":137,"threads_started":1}
I20260810 18:59:08.067968 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3f66226c341f9bea384d12e70695d" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048" is_pre_election: true
I20260810 18:59:08.068106 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3f66226c341f9bea384d12e70695d" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e22c4a7f264aba99136dfb225630d6" is_pre_election: true
I20260810 18:59:08.068300 7558 raft_consensus.cc:2454] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 0.
I20260810 18:59:08.068411 7773 raft_consensus.cc:2454] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 0.
I20260810 18:59:08.068846 7252 leader_election.cc:302] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, aee157df2c2045158ead0ec9b2f7ea7e; no voters:
I20260810 18:59:08.069126 7891 raft_consensus.cc:2781] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:08.069221 7891 raft_consensus.cc:482] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:08.069278 7891 raft_consensus.cc:3037] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.070354 7891 raft_consensus.cc:504] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:08.070703 7891 leader_election.cc:288] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 election: Requested vote from peers b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751), 09050963869d44b6a9e7268448139048 (127.27.147.130:43639)
I20260810 18:59:08.071125 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3f66226c341f9bea384d12e70695d" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048"
I20260810 18:59:08.071101 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19d3f66226c341f9bea384d12e70695d" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e22c4a7f264aba99136dfb225630d6"
I20260810 18:59:08.071239 7558 raft_consensus.cc:3037] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.071240 7773 raft_consensus.cc:3037] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.072576 7558 raft_consensus.cc:2454] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 1.
I20260810 18:59:08.072899 7252 leader_election.cc:302] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, aee157df2c2045158ead0ec9b2f7ea7e; no voters:
I20260810 18:59:08.072937 7773 raft_consensus.cc:2454] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 1.
I20260810 18:59:08.073086 7891 raft_consensus.cc:2781] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:08.073297 7891 raft_consensus.cc:686] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 1 LEADER]: Becoming Leader. State: Replica: aee157df2c2045158ead0ec9b2f7ea7e, State: Running, Role: LEADER
I20260810 18:59:08.073645 7891 consensus_queue.cc:237] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } }
I20260810 18:59:08.077322 7123 catalog_manager.cc:5495] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e reported cstate change: term changed from 0 to 1, leader changed from <none> to aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129). New cstate: current_term: 1 leader_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } health_report { overall_health: UNKNOWN } } }
I20260810 18:59:08.093608 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.090002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51246 (local address 127.27.147.129:33651)
0810 18:59:08.090143 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.090147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.090162 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:59:08.090334 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.090337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.090396 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.090501 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.090764 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.091322 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.092199 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.092424 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.092532 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.092821 (+ 289us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.093209 (+ 388us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.093334 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.093417 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.093472 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:59:08.099126 7773 raft_consensus.cc:1261] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Refusing update from remote peer aee157df2c2045158ead0ec9b2f7ea7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.099184 7558 raft_consensus.cc:1261] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Refusing update from remote peer aee157df2c2045158ead0ec9b2f7ea7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.099855 7891 consensus_queue.cc:1035] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.100059 7897 consensus_queue.cc:1035] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.110471 7907 mvcc.cc:205] Tried to move back new op lower bound from 7317046673807327232 to 7317046673712685056. Current Snapshot: MvccSnapshot[applied={T|T < 7317046673807327232 or (T in {7317046673807327232})}]
I20260810 18:59:08.115576 7125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.115742 7125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.118083 7125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Sending DeleteTablet for 3 replicas of tablet 19d3f66226c341f9bea384d12e70695d
I20260810 18:59:08.118872 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d3f66226c341f9bea384d12e70695d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:55122
I20260810 18:59:08.118881 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d3f66226c341f9bea384d12e70695d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:41146
I20260810 18:59:08.119287 7914 tablet_replica.cc:330] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: stopping tablet replica
I20260810 18:59:08.119367 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 19d3f66226c341f9bea384d12e70695d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:51230
I20260810 18:59:08.119582 7914 raft_consensus.cc:2229] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.119845 7125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59442:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 18:59:08.119894 7914 raft_consensus.cc:2258] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 18:59:08.120236 7125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:08.121057 7915 tablet_replica.cc:330] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: stopping tablet replica
I20260810 18:59:08.121341 7915 raft_consensus.cc:2229] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.121640 7915 raft_consensus.cc:2258] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.121784 7914 ts_tablet_manager.cc:1905] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.124550 7915 ts_tablet_manager.cc:1905] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.125293 7914 ts_tablet_manager.cc:1918] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:08.125326 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc587873cf3449d96cdd2e739485f62 (DEFAULT_TABLE table=test-table [id=67ce20f46e4c405dadc68909702648c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.125391 7914 log.cc:1196] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/19d3f66226c341f9bea384d12e70695d
I20260810 18:59:08.125578 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc587873cf3449d96cdd2e739485f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.125698 7914 ts_tablet_manager.cc:1939] T 19d3f66226c341f9bea384d12e70695d P 09050963869d44b6a9e7268448139048: Deleting consensus metadata
I20260810 18:59:08.126986 7917 tablet_replica.cc:330] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: stopping tablet replica
I20260810 18:59:08.127259 7917 raft_consensus.cc:2229] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:59:08.127504 7915 ts_tablet_manager.cc:1918] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:08.127596 7915 log.cc:1196] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/19d3f66226c341f9bea384d12e70695d
I20260810 18:59:08.127655 7917 raft_consensus.cc:2258] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.127907 7915 ts_tablet_manager.cc:1939] T 19d3f66226c341f9bea384d12e70695d P b0e22c4a7f264aba99136dfb225630d6: Deleting consensus metadata
I20260810 18:59:08.127929 7103 catalog_manager.cc:4841] TS 09050963869d44b6a9e7268448139048 (127.27.147.130:43639): tablet 19d3f66226c341f9bea384d12e70695d (table test-table [id=eb8e3feccb4f4673a293d2030b688677]) successfully deleted
I20260810 18:59:08.128135 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc587873cf3449d96cdd2e739485f62 (DEFAULT_TABLE table=test-table [id=67ce20f46e4c405dadc68909702648c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.128391 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc587873cf3449d96cdd2e739485f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.128399 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 2bc587873cf3449d96cdd2e739485f62 (DEFAULT_TABLE table=test-table [id=67ce20f46e4c405dadc68909702648c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.128616 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2bc587873cf3449d96cdd2e739485f62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.128911 7103 catalog_manager.cc:4841] TS b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751): tablet 19d3f66226c341f9bea384d12e70695d (table test-table [id=eb8e3feccb4f4673a293d2030b688677]) successfully deleted
I20260810 18:59:08.130643 7883 tablet_bootstrap.cc:492] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: Bootstrap starting.
I20260810 18:59:08.130686 7885 tablet_bootstrap.cc:492] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: Bootstrap starting.
I20260810 18:59:08.130708 7882 tablet_bootstrap.cc:492] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: Bootstrap starting.
I20260810 18:59:08.131596 7882 tablet_bootstrap.cc:654] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.131633 7885 tablet_bootstrap.cc:654] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.131633 7883 tablet_bootstrap.cc:654] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.133176 7882 tablet_bootstrap.cc:492] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: No bootstrap required, opened a new log
I20260810 18:59:08.133268 7882 ts_tablet_manager.cc:1397] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:08.133437 7883 tablet_bootstrap.cc:492] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: No bootstrap required, opened a new log
I20260810 18:59:08.133517 7883 ts_tablet_manager.cc:1397] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:08.133769 7885 tablet_bootstrap.cc:492] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: No bootstrap required, opened a new log
I20260810 18:59:08.133852 7885 ts_tablet_manager.cc:1397] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:08.133790 7882 raft_consensus.cc:348] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.133924 7882 raft_consensus.cc:374] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.133961 7882 raft_consensus.cc:729] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee157df2c2045158ead0ec9b2f7ea7e, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.134085 7882 consensus_queue.cc:260] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.134233 7883 raft_consensus.cc:348] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.134333 7883 raft_consensus.cc:374] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.134356 7882 ts_tablet_manager.cc:1428] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260810 18:59:08.134366 7883 raft_consensus.cc:729] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09050963869d44b6a9e7268448139048, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.134327 7885 raft_consensus.cc:348] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.134433 7885 raft_consensus.cc:374] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.134466 7885 raft_consensus.cc:729] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e22c4a7f264aba99136dfb225630d6, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.134482 7883 consensus_queue.cc:260] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.134585 7885 consensus_queue.cc:260] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.134747 7883 ts_tablet_manager.cc:1428] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.134846 7885 ts_tablet_manager.cc:1428] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.135667 7917 ts_tablet_manager.cc:1905] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.138147 7917 ts_tablet_manager.cc:1918] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:08.138270 7917 log.cc:1196] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/19d3f66226c341f9bea384d12e70695d
I20260810 18:59:08.138646 7917 ts_tablet_manager.cc:1939] T 19d3f66226c341f9bea384d12e70695d P aee157df2c2045158ead0ec9b2f7ea7e: Deleting consensus metadata
I20260810 18:59:08.139771 7104 catalog_manager.cc:4841] TS aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651): tablet 19d3f66226c341f9bea384d12e70695d (table test-table [id=eb8e3feccb4f4673a293d2030b688677]) successfully deleted
I20260810 18:59:08.334774 7893 raft_consensus.cc:482] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:08.334946 7893 raft_consensus.cc:504] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.336108 7893 leader_election.cc:288] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651), 09050963869d44b6a9e7268448139048 (127.27.147.130:43639)
I20260810 18:59:08.341580 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.336489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:50865 (local address 127.27.147.130:43639)
0810 18:59:08.336689 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.336693 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.336774 (+ 81us) server_negotiation.cc:407] Connection header received
0810 18:59:08.336936 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.336939 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.336992 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.337080 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:08.337502 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.338296 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.340232 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.340835 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.340964 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.341018 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.341373 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.341447 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 18:59:08.341580 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.336465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:57107 (local address 127.27.147.129:33651)
0810 18:59:08.336607 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.336612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.336776 (+ 164us) server_negotiation.cc:407] Connection header received
0810 18:59:08.336937 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.336940 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.336992 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.337080 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:08.337500 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.338296 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.340232 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.340835 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.340964 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.341018 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.341376 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.341448 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 18:59:08.341660 7924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.336358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43639 (local address 127.27.147.131:50865)
0810 18:59:08.336685 (+ 327us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.336700 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.336819 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.337109 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.337112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.337138 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:08.337380 (+ 242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.337388 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.338417 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.338420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.340085 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.340096 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.340963 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.340967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.341187 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.341254 (+ 67us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.341332 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.341480 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"mutex_wait_us":294,"spinlock_wait_cycles":28672,"thread_start_us":91,"threads_started":1}
I20260810 18:59:08.341665 7925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.336342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:33651 (local address 127.27.147.131:57107)
0810 18:59:08.336665 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.336680 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.336819 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.337109 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.337112 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.337138 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:08.337381 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.337388 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.338417 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.338420 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.340084 (+ 1664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.340096 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.340969 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.340972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.341181 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.341256 (+ 75us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.341332 (+ 76us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.341480 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":250,"mutex_wait_us":410,"spinlock_wait_cycles":19328,"thread_start_us":108,"threads_started":1}
I20260810 18:59:08.342109 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc587873cf3449d96cdd2e739485f62" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048" is_pre_election: true
I20260810 18:59:08.342293 7558 raft_consensus.cc:2454] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 0.
I20260810 18:59:08.342260 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc587873cf3449d96cdd2e739485f62" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" is_pre_election: true
I20260810 18:59:08.342523 7360 raft_consensus.cc:2454] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 0.
I20260810 18:59:08.342733 7670 leader_election.cc:302] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, b0e22c4a7f264aba99136dfb225630d6; no voters:
I20260810 18:59:08.343024 7893 raft_consensus.cc:2781] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:08.343091 7893 raft_consensus.cc:482] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:08.343128 7893 raft_consensus.cc:3037] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.344275 7893 raft_consensus.cc:504] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.344638 7893 leader_election.cc:288] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 election: Requested vote from peers aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651), 09050963869d44b6a9e7268448139048 (127.27.147.130:43639)
I20260810 18:59:08.344971 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc587873cf3449d96cdd2e739485f62" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee157df2c2045158ead0ec9b2f7ea7e"
I20260810 18:59:08.344993 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2bc587873cf3449d96cdd2e739485f62" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048"
I20260810 18:59:08.345099 7558 raft_consensus.cc:3037] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.345099 7360 raft_consensus.cc:3037] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.346242 7360 raft_consensus.cc:2454] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 1.
I20260810 18:59:08.346242 7558 raft_consensus.cc:2454] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 1.
I20260810 18:59:08.346529 7670 leader_election.cc:302] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, aee157df2c2045158ead0ec9b2f7ea7e, b0e22c4a7f264aba99136dfb225630d6; no voters:
I20260810 18:59:08.346725 7893 raft_consensus.cc:2781] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:08.346917 7893 raft_consensus.cc:686] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 1 LEADER]: Becoming Leader. State: Replica: b0e22c4a7f264aba99136dfb225630d6, State: Running, Role: LEADER
I20260810 18:59:08.347177 7893 consensus_queue.cc:237] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.349503 7123 catalog_manager.cc:5495] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131). New cstate: current_term: 1 leader_uuid: "b0e22c4a7f264aba99136dfb225630d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } health_report { overall_health: HEALTHY } } }
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388346, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.190@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:08.384635 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.376425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38065 (local address 127.0.0.1:59458)
0810 18:59:08.376712 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.376731 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.376829 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.377124 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.377127 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.377639 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:08.377740 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.377746 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.378497 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.378501 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.379273 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.379289 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.379732 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.379735 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.379796 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.380340 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:08.380352 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:08.381733 (+ 1381us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:08.383301 (+ 1568us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:08.383310 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:08.383316 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:08.383547 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:08.383789 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:08.383792 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:08.383793 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:08.383877 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:08.384197 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:08.384202 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:08.384302 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.384395 (+ 93us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.384500 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":117,"threads_started":1}
I20260810 18:59:08.384665 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.376504 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59458 (local address 127.27.147.190:38065)
0810 18:59:08.376737 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.376744 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.376785 (+ 41us) server_negotiation.cc:407] Connection header received
0810 18:59:08.376909 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.376912 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.376954 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.377038 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:08.377852 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.378382 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.379403 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.379634 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.379709 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.381859 (+ 2150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:08.381862 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:08.381864 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:08.381885 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:08.383185 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:08.383632 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:08.383635 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:08.383636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:08.383710 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:08.383953 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:08.383955 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:08.383956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:08.384110 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:08.384182 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.384448 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.384547 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":99,"threads_started":1}
I20260810 18:59:08.388362 7123 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:59458:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 18:59:08.389002 7123 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:08.393533 7328 tablet_service.cc:1467] Processing CreateTablet for tablet d07dd0bf479549aab38eb8c320e950ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc0d38c998ad4bff96060172a47b5b76]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:59:08.393566 7749 tablet_service.cc:1467] Processing CreateTablet for tablet d07dd0bf479549aab38eb8c320e950ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc0d38c998ad4bff96060172a47b5b76]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:59:08.393558 7540 tablet_service.cc:1467] Processing CreateTablet for tablet d07dd0bf479549aab38eb8c320e950ea (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=fc0d38c998ad4bff96060172a47b5b76]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 18:59:08.393930 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07dd0bf479549aab38eb8c320e950ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.393930 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07dd0bf479549aab38eb8c320e950ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.393975 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d07dd0bf479549aab38eb8c320e950ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.396737 7882 tablet_bootstrap.cc:492] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: Bootstrap starting.
I20260810 18:59:08.396756 7885 tablet_bootstrap.cc:492] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: Bootstrap starting.
I20260810 18:59:08.396903 7883 tablet_bootstrap.cc:492] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: Bootstrap starting.
I20260810 18:59:08.397629 7882 tablet_bootstrap.cc:654] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.397629 7885 tablet_bootstrap.cc:654] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.397775 7883 tablet_bootstrap.cc:654] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.399140 7882 tablet_bootstrap.cc:492] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: No bootstrap required, opened a new log
I20260810 18:59:08.399240 7882 ts_tablet_manager.cc:1397] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:08.399747 7882 raft_consensus.cc:348] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.399887 7882 raft_consensus.cc:374] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.399926 7882 raft_consensus.cc:729] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee157df2c2045158ead0ec9b2f7ea7e, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.400084 7882 consensus_queue.cc:260] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.400449 7882 ts_tablet_manager.cc:1428] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.400457 7891 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.400990 7891 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:59:08.404045 7883 tablet_bootstrap.cc:492] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: No bootstrap required, opened a new log
I20260810 18:59:08.404047 7885 tablet_bootstrap.cc:492] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: No bootstrap required, opened a new log
I20260810 18:59:08.404134 7883 ts_tablet_manager.cc:1397] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 18:59:08.404134 7885 ts_tablet_manager.cc:1397] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260810 18:59:08.404570 7885 raft_consensus.cc:348] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.404593 7883 raft_consensus.cc:348] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.404686 7883 raft_consensus.cc:374] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.404685 7885 raft_consensus.cc:374] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.404719 7883 raft_consensus.cc:729] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09050963869d44b6a9e7268448139048, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.404744 7885 raft_consensus.cc:729] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e22c4a7f264aba99136dfb225630d6, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.404863 7883 consensus_queue.cc:260] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.404863 7885 consensus_queue.cc:260] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.405138 7885 ts_tablet_manager.cc:1428] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.405138 7883 ts_tablet_manager.cc:1428] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.405339 7893 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.405341 7892 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.405443 7893 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:59:08.405489 7892 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:59:08.436247 7892 raft_consensus.cc:482] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:08.436367 7892 raft_consensus.cc:504] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.437430 7892 leader_election.cc:288] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651), b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751)
I20260810 18:59:08.443020 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.437823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:49689 (local address 127.27.147.131:38751)
0810 18:59:08.437995 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.438000 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.438260 (+ 260us) server_negotiation.cc:407] Connection header received
0810 18:59:08.438414 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.438420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.438490 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.438578 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:08.439001 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.439783 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.441712 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.442312 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.442435 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.442487 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.442776 (+ 289us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.442890 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 18:59:08.443018 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.437817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:49985 (local address 127.27.147.129:33651)
0810 18:59:08.437995 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.438002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.438383 (+ 381us) server_negotiation.cc:407] Connection header received
0810 18:59:08.438492 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.438495 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.438549 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.438623 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:08.439000 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.439783 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.441712 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.442312 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.442435 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.442489 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.442796 (+ 307us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.442899 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 18:59:08.443053 7945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.437705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:38751 (local address 127.27.147.130:49689)
0810 18:59:08.438164 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.438181 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.438298 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.438616 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.438622 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.438642 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:08.438877 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.438885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.439902 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.439906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.441581 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.441593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.442437 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.442441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.442619 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.442664 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.442779 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.442885 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"mutex_wait_us":218,"spinlock_wait_cycles":8448,"thread_start_us":150,"threads_started":1}
I20260810 18:59:08.443450 7943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.437705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:33651 (local address 127.27.147.130:49985)
0810 18:59:08.438271 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.438283 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.438382 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.438643 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.438646 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.438657 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:08.438877 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.438885 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.439902 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.439906 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.441581 (+ 1675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.441593 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.442437 (+ 844us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.442441 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.442622 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.442668 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.443220 (+ 552us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.443312 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"mutex_wait_us":347,"spinlock_wait_cycles":2944,"thread_start_us":123,"threads_started":1}
I20260810 18:59:08.443435 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07dd0bf479549aab38eb8c320e950ea" candidate_uuid: "09050963869d44b6a9e7268448139048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e22c4a7f264aba99136dfb225630d6" is_pre_election: true
I20260810 18:59:08.443614 7773 raft_consensus.cc:2454] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09050963869d44b6a9e7268448139048 in term 0.
I20260810 18:59:08.443969 7462 leader_election.cc:302] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, b0e22c4a7f264aba99136dfb225630d6; no voters:
I20260810 18:59:08.444007 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07dd0bf479549aab38eb8c320e950ea" candidate_uuid: "09050963869d44b6a9e7268448139048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" is_pre_election: true
I20260810 18:59:08.444134 7360 raft_consensus.cc:2454] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 09050963869d44b6a9e7268448139048 in term 0.
I20260810 18:59:08.444432 7892 raft_consensus.cc:2781] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:08.444522 7892 raft_consensus.cc:482] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:08.444585 7892 raft_consensus.cc:3037] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.445708 7892 raft_consensus.cc:504] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.445990 7892 leader_election.cc:288] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [CANDIDATE]: Term 1 election: Requested vote from peers aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651), b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751)
I20260810 18:59:08.446322 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07dd0bf479549aab38eb8c320e950ea" candidate_uuid: "09050963869d44b6a9e7268448139048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee157df2c2045158ead0ec9b2f7ea7e"
I20260810 18:59:08.446472 7360 raft_consensus.cc:3037] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.446964 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d07dd0bf479549aab38eb8c320e950ea" candidate_uuid: "09050963869d44b6a9e7268448139048" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e22c4a7f264aba99136dfb225630d6"
I20260810 18:59:08.447057 7773 raft_consensus.cc:3037] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.447527 7360 raft_consensus.cc:2454] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09050963869d44b6a9e7268448139048 in term 1.
I20260810 18:59:08.447817 7464 leader_election.cc:302] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, aee157df2c2045158ead0ec9b2f7ea7e; no voters:
I20260810 18:59:08.447984 7892 raft_consensus.cc:2781] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:08.448169 7773 raft_consensus.cc:2454] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 09050963869d44b6a9e7268448139048 in term 1.
I20260810 18:59:08.448225 7892 raft_consensus.cc:686] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [term 1 LEADER]: Becoming Leader. State: Replica: 09050963869d44b6a9e7268448139048, State: Running, Role: LEADER
I20260810 18:59:08.448496 7892 consensus_queue.cc:237] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.449671 7946 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: TxnStatusTablet state changed. Reason: New leader 09050963869d44b6a9e7268448139048. Latest consensus state: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.449790 7946 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:59:08.450047 7948 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 18:59:08.450232 7948 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 18:59:08.450935 7123 catalog_manager.cc:5495] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 reported cstate change: term changed from 0 to 1, leader changed from <none> to 09050963869d44b6a9e7268448139048 (127.27.147.130). New cstate: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20260810 18:59:08.464442 7936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.460404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43639 (local address 127.0.0.1:55126)
0810 18:59:08.460560 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.460575 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.460663 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.461001 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.461003 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.461010 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:59:08.461097 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.461102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.462081 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.462084 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.462747 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.462756 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.463392 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.463394 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.463469 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.463514 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.464217 (+ 703us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.464229 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.464278 (+ 49us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.464343 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 18:59:08.464463 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.460524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.27.147.130:43639)
0810 18:59:08.460626 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.460630 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.460643 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:08.460791 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.460795 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.460889 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.460990 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.461215 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.461985 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.462900 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.463264 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.463404 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.463661 (+ 257us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.464126 (+ 465us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.464235 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.464292 (+ 57us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.464349 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:59:08.467248 7773 raft_consensus.cc:1261] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Refusing update from remote peer 09050963869d44b6a9e7268448139048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.467547 7360 raft_consensus.cc:1261] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Refusing update from remote peer 09050963869d44b6a9e7268448139048: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.467679 7892 consensus_queue.cc:1035] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.468068 7892 consensus_queue.cc:1035] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.469132 7893 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: TxnStatusTablet state changed. Reason: New leader 09050963869d44b6a9e7268448139048. Latest consensus state: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.469236 7893 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:59:08.470587 7891 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: TxnStatusTablet state changed. Reason: New leader 09050963869d44b6a9e7268448139048. Latest consensus state: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.470708 7891 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:59:08.471719 7892 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.471747 7946 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.471818 7892 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:59:08.471844 7946 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P 09050963869d44b6a9e7268448139048: This TxnStatusTablet replica's current role is: LEADER
I20260810 18:59:08.472895 7893 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.472988 7893 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P b0e22c4a7f264aba99136dfb225630d6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:59:08.473619 7904 mvcc.cc:205] Tried to move back new op lower bound from 7317046675316932608 to 7317046675247280128. Current Snapshot: MvccSnapshot[applied={T|T < 7317046675316932608 or (T in {7317046675316932608})}]
I20260810 18:59:08.475845 7891 tablet_replica.cc:439] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "09050963869d44b6a9e7268448139048" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } } }
I20260810 18:59:08.475952 7891 tablet_replica.cc:442] T d07dd0bf479549aab38eb8c320e950ea P aee157df2c2045158ead0ec9b2f7ea7e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 18:59:08.481135 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.477367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41152 (local address 127.27.147.131:38751)
0810 18:59:08.477488 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.477495 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.477508 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:08.477576 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.477579 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.477646 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.477745 (+ 99us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.477992 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.478503 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.479668 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.479872 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.479949 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.480274 (+ 325us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.480720 (+ 446us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.480835 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.480921 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.480971 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:59:08.492237 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.488359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:40711 (local address 127.27.147.130:43639)
0810 18:59:08.488480 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.488485 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.488649 (+ 164us) server_negotiation.cc:407] Connection header received
0810 18:59:08.488789 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.488792 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.488829 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.488908 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.489161 (+ 253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.489628 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.491178 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.491378 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.491503 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.491795 (+ 292us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.491858 (+ 63us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.491940 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.492042 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.492105 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260810 18:59:08.492254 7958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.488276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43639 (local address 127.27.147.131:40711)
0810 18:59:08.488573 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.488584 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.488692 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.488917 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.488920 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.488928 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:59:08.489061 (+ 133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.489066 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.489735 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.489740 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.490979 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.490990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.491521 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.491525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.491619 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.491657 (+ 38us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.491947 (+ 290us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.491963 (+ 16us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.492021 (+ 58us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.492100 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"thread_start_us":93,"threads_started":1}
I20260810 18:59:08.498668 7360 raft_consensus.cc:1261] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Refusing update from remote peer b0e22c4a7f264aba99136dfb225630d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.499119 7926 consensus_queue.cc:1035] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.501214 7558 raft_consensus.cc:1261] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Refusing update from remote peer b0e22c4a7f264aba99136dfb225630d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.501695 7926 consensus_queue.cc:1035] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.508618 7902 mvcc.cc:205] Tried to move back new op lower bound from 7317046675446046720 to 7317046674832588800. Current Snapshot: MvccSnapshot[applied={T|T < 7317046675446046720}]
I20260810 18:59:08.530243 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.523960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:56701 (local address 127.27.147.131:38751)
0810 18:59:08.524063 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.524067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.527166 (+ 3099us) server_negotiation.cc:407] Connection header received
0810 18:59:08.527296 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.527300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.527342 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.527434 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.527692 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.528202 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.529065 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.529304 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.529413 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.529721 (+ 308us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.529802 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.529877 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.530049 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.530108 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 18:59:08.530263 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.523655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:38751 (local address 127.27.147.130:56701)
0810 18:59:08.527084 (+ 3429us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.527101 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.527212 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.527446 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.527449 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.527464 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:59:08.527579 (+ 115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.527589 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.528315 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.528320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.528943 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.528951 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.529416 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.529420 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.529547 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.529601 (+ 54us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.529900 (+ 299us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.529908 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.530020 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.530099 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3299,"thread_start_us":91,"threads_started":1}
I20260810 18:59:08.556222 7121 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.556399 7121 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59442:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.558696 7121 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Sending DeleteTablet for 3 replicas of tablet 2bc587873cf3449d96cdd2e739485f62
I20260810 18:59:08.559298 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc587873cf3449d96cdd2e739485f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:51230
I20260810 18:59:08.559404 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc587873cf3449d96cdd2e739485f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:41146
I20260810 18:59:08.559517 7917 tablet_replica.cc:330] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: stopping tablet replica
I20260810 18:59:08.559569 7915 tablet_replica.cc:330] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: stopping tablet replica
I20260810 18:59:08.559638 7917 raft_consensus.cc:2229] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.559666 7915 raft_consensus.cc:2229] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:59:08.559747 7917 raft_consensus.cc:2258] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.559947 7915 raft_consensus.cc:2258] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.560217 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 2bc587873cf3449d96cdd2e739485f62 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:55122
I20260810 18:59:08.560407 7914 tablet_replica.cc:330] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: stopping tablet replica
I20260810 18:59:08.560529 7914 raft_consensus.cc:2229] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.560643 7914 raft_consensus.cc:2258] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.560745 7915 ts_tablet_manager.cc:1905] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.563577 7914 ts_tablet_manager.cc:1905] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.565091 7917 ts_tablet_manager.cc:1905] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.566304 7914 ts_tablet_manager.cc:1918] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:59:08.566388 7914 log.cc:1196] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/2bc587873cf3449d96cdd2e739485f62
I20260810 18:59:08.566668 7914 ts_tablet_manager.cc:1939] T 2bc587873cf3449d96cdd2e739485f62 P 09050963869d44b6a9e7268448139048: Deleting consensus metadata
I20260810 18:59:08.567358 7103 catalog_manager.cc:4841] TS 09050963869d44b6a9e7268448139048 (127.27.147.130:43639): tablet 2bc587873cf3449d96cdd2e739485f62 (table test-table [id=67ce20f46e4c405dadc68909702648c5]) successfully deleted
I20260810 18:59:08.567915 7917 ts_tablet_manager.cc:1918] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:59:08.568002 7917 log.cc:1196] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/2bc587873cf3449d96cdd2e739485f62
I20260810 18:59:08.568265 7917 ts_tablet_manager.cc:1939] T 2bc587873cf3449d96cdd2e739485f62 P aee157df2c2045158ead0ec9b2f7ea7e: Deleting consensus metadata
I20260810 18:59:08.568639 7915 ts_tablet_manager.cc:1918] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:59:08.568722 7915 log.cc:1196] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/2bc587873cf3449d96cdd2e739485f62
I20260810 18:59:08.568992 7915 ts_tablet_manager.cc:1939] T 2bc587873cf3449d96cdd2e739485f62 P b0e22c4a7f264aba99136dfb225630d6: Deleting consensus metadata
I20260810 18:59:08.568994 7104 catalog_manager.cc:4841] TS aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651): tablet 2bc587873cf3449d96cdd2e739485f62 (table test-table [id=67ce20f46e4c405dadc68909702648c5]) successfully deleted
I20260810 18:59:08.569671 7103 catalog_manager.cc:4841] TS b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751): tablet 2bc587873cf3449d96cdd2e739485f62 (table test-table [id=67ce20f46e4c405dadc68909702648c5]) successfully deleted
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[7061](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:59:08.600246 7938 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:08.592052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59468 (local address 127.27.147.190:38065)
0810 18:59:08.592190 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.592195 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.592496 (+ 301us) server_negotiation.cc:407] Connection header received
0810 18:59:08.592645 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.592649 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.592714 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.592815 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:08.593716 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.594578 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.595840 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.596188 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.596309 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.600107 (+ 3798us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59468: BlockingRecv error: recv got EOF from 127.0.0.1:59468 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[7061](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM
I20260810 18:59:08.612663 7938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.602185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59478 (local address 127.27.147.190:38065)
0810 18:59:08.602295 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.602300 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.602741 (+ 441us) server_negotiation.cc:407] Connection header received
0810 18:59:08.602851 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.602855 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.602922 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.603011 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:08.603745 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.604514 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.605673 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.606016 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.606133 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.609142 (+ 3009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:08.609147 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:08.609150 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:08.609177 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:08.610953 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:08.611430 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:08.611434 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:08.611435 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:08.611526 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:08.611813 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:08.611816 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:08.611818 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:08.612016 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:08.612141 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.612383 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.612500 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 18:59:08.615372 7121 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:59:08.615821 7121 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:08.621176 7328 tablet_service.cc:1467] Processing CreateTablet for tablet 3db311c561be4812981dc47bd82f51d4 (DEFAULT_TABLE table=test-table [id=2a3c3c1fe4dd485686f0e73e4bb6f423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.621236 7540 tablet_service.cc:1467] Processing CreateTablet for tablet 3db311c561be4812981dc47bd82f51d4 (DEFAULT_TABLE table=test-table [id=2a3c3c1fe4dd485686f0e73e4bb6f423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.621390 7749 tablet_service.cc:1467] Processing CreateTablet for tablet 3db311c561be4812981dc47bd82f51d4 (DEFAULT_TABLE table=test-table [id=2a3c3c1fe4dd485686f0e73e4bb6f423]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.621512 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db311c561be4812981dc47bd82f51d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.621567 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db311c561be4812981dc47bd82f51d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.621680 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3db311c561be4812981dc47bd82f51d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.624060 7882 tablet_bootstrap.cc:492] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: Bootstrap starting.
I20260810 18:59:08.624119 7883 tablet_bootstrap.cc:492] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: Bootstrap starting.
I20260810 18:59:08.624364 7885 tablet_bootstrap.cc:492] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: Bootstrap starting.
I20260810 18:59:08.625011 7882 tablet_bootstrap.cc:654] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.625013 7883 tablet_bootstrap.cc:654] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.625319 7885 tablet_bootstrap.cc:654] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.626582 7883 tablet_bootstrap.cc:492] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: No bootstrap required, opened a new log
I20260810 18:59:08.626683 7883 ts_tablet_manager.cc:1397] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:08.627172 7885 tablet_bootstrap.cc:492] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: No bootstrap required, opened a new log
I20260810 18:59:08.627158 7883 raft_consensus.cc:348] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.627244 7885 ts_tablet_manager.cc:1397] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 18:59:08.627264 7883 raft_consensus.cc:374] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.627293 7883 raft_consensus.cc:729] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09050963869d44b6a9e7268448139048, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.627418 7883 consensus_queue.cc:260] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.627661 7883 ts_tablet_manager.cc:1428] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.627686 7885 raft_consensus.cc:348] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.627784 7885 raft_consensus.cc:374] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.627813 7885 raft_consensus.cc:729] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e22c4a7f264aba99136dfb225630d6, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.627925 7885 consensus_queue.cc:260] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.628180 7885 ts_tablet_manager.cc:1428] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:59:08.630342 7882 tablet_bootstrap.cc:492] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: No bootstrap required, opened a new log
I20260810 18:59:08.630424 7882 ts_tablet_manager.cc:1397] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260810 18:59:08.630782 7882 raft_consensus.cc:348] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.630920 7882 raft_consensus.cc:374] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.630954 7882 raft_consensus.cc:729] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee157df2c2045158ead0ec9b2f7ea7e, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.631079 7882 consensus_queue.cc:260] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.631330 7882 ts_tablet_manager.cc:1428] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.636626 7926 raft_consensus.cc:482] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:08.636713 7926 raft_consensus.cc:504] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.637076 7926 leader_election.cc:288] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651), 09050963869d44b6a9e7268448139048 (127.27.147.130:43639)
I20260810 18:59:08.637377 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db311c561be4812981dc47bd82f51d4" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" is_pre_election: true
I20260810 18:59:08.637495 7360 raft_consensus.cc:2454] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 0.
I20260810 18:59:08.637506 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db311c561be4812981dc47bd82f51d4" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048" is_pre_election: true
I20260810 18:59:08.637624 7558 raft_consensus.cc:2454] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 0.
I20260810 18:59:08.637838 7671 leader_election.cc:302] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aee157df2c2045158ead0ec9b2f7ea7e, b0e22c4a7f264aba99136dfb225630d6; no voters:
I20260810 18:59:08.638000 7926 raft_consensus.cc:2781] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:08.638057 7926 raft_consensus.cc:482] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:08.638099 7926 raft_consensus.cc:3037] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.639173 7926 raft_consensus.cc:504] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.639495 7926 leader_election.cc:288] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 election: Requested vote from peers aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651), 09050963869d44b6a9e7268448139048 (127.27.147.130:43639)
I20260810 18:59:08.639776 7360 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db311c561be4812981dc47bd82f51d4" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aee157df2c2045158ead0ec9b2f7ea7e"
I20260810 18:59:08.639873 7360 raft_consensus.cc:3037] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.639869 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3db311c561be4812981dc47bd82f51d4" candidate_uuid: "b0e22c4a7f264aba99136dfb225630d6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048"
I20260810 18:59:08.639963 7558 raft_consensus.cc:3037] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.640897 7360 raft_consensus.cc:2454] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 1.
I20260810 18:59:08.640906 7558 raft_consensus.cc:2454] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b0e22c4a7f264aba99136dfb225630d6 in term 1.
I20260810 18:59:08.641203 7670 leader_election.cc:302] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, b0e22c4a7f264aba99136dfb225630d6; no voters:
I20260810 18:59:08.641377 7926 raft_consensus.cc:2781] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:08.641489 7926 raft_consensus.cc:686] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 1 LEADER]: Becoming Leader. State: Replica: b0e22c4a7f264aba99136dfb225630d6, State: Running, Role: LEADER
I20260810 18:59:08.641657 7926 consensus_queue.cc:237] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.643308 7125 catalog_manager.cc:5495] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 reported cstate change: term changed from 0 to 1, leader changed from <none> to b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131). New cstate: current_term: 1 leader_uuid: "b0e22c4a7f264aba99136dfb225630d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } health_report { overall_health: HEALTHY } } }
I20260810 18:59:08.653383 7874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.649987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41156 (local address 127.27.147.131:38751)
0810 18:59:08.650107 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.650111 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.650144 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:59:08.650286 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.650288 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.650339 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.650401 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.650675 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.651224 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.652424 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.652627 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.652695 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.652983 (+ 288us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.653044 (+ 61us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.653096 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.653217 (+ 121us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.653260 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 18:59:08.655057 7558 raft_consensus.cc:1261] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Refusing update from remote peer b0e22c4a7f264aba99136dfb225630d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.655069 7360 raft_consensus.cc:1261] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Refusing update from remote peer b0e22c4a7f264aba99136dfb225630d6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.655481 7966 consensus_queue.cc:1035] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.655682 7926 consensus_queue.cc:1035] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.664635 7125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.664778 7125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.666707 7125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Sending DeleteTablet for 3 replicas of tablet 3db311c561be4812981dc47bd82f51d4
I20260810 18:59:08.667273 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db311c561be4812981dc47bd82f51d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:51230
I20260810 18:59:08.667433 7917 tablet_replica.cc:330] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: stopping tablet replica
I20260810 18:59:08.667500 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db311c561be4812981dc47bd82f51d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:55122
I20260810 18:59:08.667541 7917 raft_consensus.cc:2229] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.667615 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet 3db311c561be4812981dc47bd82f51d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:41146
I20260810 18:59:08.667654 7917 raft_consensus.cc:2258] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.667770 7915 tablet_replica.cc:330] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: stopping tablet replica
I20260810 18:59:08.667797 7914 tablet_replica.cc:330] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: stopping tablet replica
I20260810 18:59:08.667896 7914 raft_consensus.cc:2229] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.667905 7915 raft_consensus.cc:2229] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:59:08.668012 7914 raft_consensus.cc:2258] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.668103 7915 raft_consensus.cc:2258] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.668754 7915 ts_tablet_manager.cc:1905] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.668955 7914 ts_tablet_manager.cc:1905] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.670681 7915 ts_tablet_manager.cc:1918] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:08.670778 7915 log.cc:1196] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/3db311c561be4812981dc47bd82f51d4
I20260810 18:59:08.671064 7915 ts_tablet_manager.cc:1939] T 3db311c561be4812981dc47bd82f51d4 P b0e22c4a7f264aba99136dfb225630d6: Deleting consensus metadata
I20260810 18:59:08.671355 7914 ts_tablet_manager.cc:1918] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:08.671435 7914 log.cc:1196] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/3db311c561be4812981dc47bd82f51d4
I20260810 18:59:08.671694 7914 ts_tablet_manager.cc:1939] T 3db311c561be4812981dc47bd82f51d4 P 09050963869d44b6a9e7268448139048: Deleting consensus metadata
I20260810 18:59:08.671744 7103 catalog_manager.cc:4841] TS b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751): tablet 3db311c561be4812981dc47bd82f51d4 (table test-table [id=2a3c3c1fe4dd485686f0e73e4bb6f423]) successfully deleted
I20260810 18:59:08.671793 7125 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59478:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:59:08.672183 7125 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:08.672286 7103 catalog_manager.cc:4841] TS 09050963869d44b6a9e7268448139048 (127.27.147.130:43639): tablet 3db311c561be4812981dc47bd82f51d4 (table test-table [id=2a3c3c1fe4dd485686f0e73e4bb6f423]) successfully deleted
I20260810 18:59:08.673266 7917 ts_tablet_manager.cc:1905] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.675630 7917 ts_tablet_manager.cc:1918] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:08.675709 7917 log.cc:1196] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/3db311c561be4812981dc47bd82f51d4
I20260810 18:59:08.675966 7917 ts_tablet_manager.cc:1939] T 3db311c561be4812981dc47bd82f51d4 P aee157df2c2045158ead0ec9b2f7ea7e: Deleting consensus metadata
I20260810 18:59:08.676596 7104 catalog_manager.cc:4841] TS aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651): tablet 3db311c561be4812981dc47bd82f51d4 (table test-table [id=2a3c3c1fe4dd485686f0e73e4bb6f423]) successfully deleted
I20260810 18:59:08.676642 7328 tablet_service.cc:1467] Processing CreateTablet for tablet a0734ff4b6ed49f3af236f65e5dbdeb0 (DEFAULT_TABLE table=test-table [id=67aa02ffa5ed4faf8dd65a10c4074769]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.676868 7328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0734ff4b6ed49f3af236f65e5dbdeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.677927 7540 tablet_service.cc:1467] Processing CreateTablet for tablet a0734ff4b6ed49f3af236f65e5dbdeb0 (DEFAULT_TABLE table=test-table [id=67aa02ffa5ed4faf8dd65a10c4074769]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.678193 7540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0734ff4b6ed49f3af236f65e5dbdeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.678191 7749 tablet_service.cc:1467] Processing CreateTablet for tablet a0734ff4b6ed49f3af236f65e5dbdeb0 (DEFAULT_TABLE table=test-table [id=67aa02ffa5ed4faf8dd65a10c4074769]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:08.678427 7749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a0734ff4b6ed49f3af236f65e5dbdeb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:08.679071 7882 tablet_bootstrap.cc:492] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e: Bootstrap starting.
I20260810 18:59:08.679840 7882 tablet_bootstrap.cc:654] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.680671 7883 tablet_bootstrap.cc:492] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: Bootstrap starting.
I20260810 18:59:08.681145 7882 tablet_bootstrap.cc:492] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e: No bootstrap required, opened a new log
I20260810 18:59:08.681234 7882 ts_tablet_manager.cc:1397] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260810 18:59:08.681378 7883 tablet_bootstrap.cc:654] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.681622 7882 raft_consensus.cc:348] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.681739 7882 raft_consensus.cc:374] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.681772 7882 raft_consensus.cc:729] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aee157df2c2045158ead0ec9b2f7ea7e, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.681892 7885 tablet_bootstrap.cc:492] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: Bootstrap starting.
I20260810 18:59:08.681888 7882 consensus_queue.cc:260] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.682152 7882 ts_tablet_manager.cc:1428] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 18:59:08.682653 7885 tablet_bootstrap.cc:654] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:08.683894 7883 tablet_bootstrap.cc:492] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: No bootstrap required, opened a new log
I20260810 18:59:08.683997 7883 ts_tablet_manager.cc:1397] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260810 18:59:08.684324 7883 raft_consensus.cc:348] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.684432 7883 raft_consensus.cc:374] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.684477 7883 raft_consensus.cc:729] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09050963869d44b6a9e7268448139048, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.684600 7883 consensus_queue.cc:260] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.684881 7883 ts_tablet_manager.cc:1428] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260810 18:59:08.685434 7885 tablet_bootstrap.cc:492] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: No bootstrap required, opened a new log
I20260810 18:59:08.685521 7885 ts_tablet_manager.cc:1397] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260810 18:59:08.685972 7885 raft_consensus.cc:348] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.686080 7885 raft_consensus.cc:374] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:08.686142 7885 raft_consensus.cc:729] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0e22c4a7f264aba99136dfb225630d6, State: Initialized, Role: FOLLOWER
I20260810 18:59:08.686259 7885 consensus_queue.cc:260] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.686530 7885 ts_tablet_manager.cc:1428] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260810 18:59:08.795851 7891 raft_consensus.cc:482] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:08.796000 7891 raft_consensus.cc:504] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.796420 7891 leader_election.cc:288] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09050963869d44b6a9e7268448139048 (127.27.147.130:43639), b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751)
I20260810 18:59:08.797000 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734ff4b6ed49f3af236f65e5dbdeb0" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e22c4a7f264aba99136dfb225630d6" is_pre_election: true
I20260810 18:59:08.797000 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734ff4b6ed49f3af236f65e5dbdeb0" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048" is_pre_election: true
I20260810 18:59:08.797168 7773 raft_consensus.cc:2454] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 0.
I20260810 18:59:08.797168 7558 raft_consensus.cc:2454] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 0.
I20260810 18:59:08.797498 7252 leader_election.cc:302] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, aee157df2c2045158ead0ec9b2f7ea7e; no voters:
I20260810 18:59:08.797693 7891 raft_consensus.cc:2781] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:08.797755 7891 raft_consensus.cc:482] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:08.797798 7891 raft_consensus.cc:3037] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.798802 7891 raft_consensus.cc:504] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.799217 7891 leader_election.cc:288] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 election: Requested vote from peers 09050963869d44b6a9e7268448139048 (127.27.147.130:43639), b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751)
I20260810 18:59:08.799607 7558 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734ff4b6ed49f3af236f65e5dbdeb0" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09050963869d44b6a9e7268448139048"
I20260810 18:59:08.799606 7773 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a0734ff4b6ed49f3af236f65e5dbdeb0" candidate_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0e22c4a7f264aba99136dfb225630d6"
I20260810 18:59:08.799733 7558 raft_consensus.cc:3037] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.799733 7773 raft_consensus.cc:3037] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:08.800868 7773 raft_consensus.cc:2454] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 1.
I20260810 18:59:08.800891 7558 raft_consensus.cc:2454] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aee157df2c2045158ead0ec9b2f7ea7e in term 1.
I20260810 18:59:08.801162 7252 leader_election.cc:302] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09050963869d44b6a9e7268448139048, aee157df2c2045158ead0ec9b2f7ea7e; no voters:
I20260810 18:59:08.801373 7891 raft_consensus.cc:2781] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:08.801609 7891 raft_consensus.cc:686] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 LEADER]: Becoming Leader. State: Replica: aee157df2c2045158ead0ec9b2f7ea7e, State: Running, Role: LEADER
I20260810 18:59:08.801770 7891 consensus_queue.cc:237] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } }
I20260810 18:59:08.803398 7125 catalog_manager.cc:5495] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e reported cstate change: term changed from 0 to 1, leader changed from <none> to aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129). New cstate: current_term: 1 leader_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aee157df2c2045158ead0ec9b2f7ea7e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 33651 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 } health_report { overall_health: UNKNOWN } } }
I20260810 18:59:08.820514 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.817286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51258 (local address 127.27.147.129:33651)
0810 18:59:08.817415 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.817421 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.817461 (+ 40us) server_negotiation.cc:407] Connection header received
0810 18:59:08.817623 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.817626 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.817671 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.817758 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.818010 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.818532 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.819460 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.819698 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.819812 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.820075 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.820153 (+ 78us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.820228 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.820328 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.820387 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 18:59:08.830149 8008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.826616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43639 (local address 127.27.147.129:46859)
0810 18:59:08.827067 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.827079 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.827193 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.827438 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.827441 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.827448 (+ 7us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:59:08.827542 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.827547 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.828278 (+ 731us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.828283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.828880 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.828889 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.829385 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.829389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.829482 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.829523 (+ 41us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.829848 (+ 325us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.829857 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.829926 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.830017 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":103,"threads_started":1}
I20260810 18:59:08.830143 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.826737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:46859 (local address 127.27.147.130:43639)
0810 18:59:08.826979 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.826983 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.827150 (+ 167us) server_negotiation.cc:407] Connection header received
0810 18:59:08.827298 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.827307 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.827348 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.827428 (+ 80us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.827655 (+ 227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.828174 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.829007 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.829272 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.829381 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.829663 (+ 282us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.829736 (+ 73us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.829841 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.829966 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.830016 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154}
I20260810 18:59:08.836059 7773 raft_consensus.cc:1261] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Refusing update from remote peer aee157df2c2045158ead0ec9b2f7ea7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.836123 7558 raft_consensus.cc:1261] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Refusing update from remote peer aee157df2c2045158ead0ec9b2f7ea7e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:08.836469 7891 consensus_queue.cc:1035] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0e22c4a7f264aba99136dfb225630d6" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 38751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.836676 8002 consensus_queue.cc:1035] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [LEADER]: Connected to new peer: Peer: permanent_uuid: "09050963869d44b6a9e7268448139048" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:08.852754 7877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.849776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:37403 (local address 127.27.147.129:33651)
0810 18:59:08.849903 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:08.849907 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:08.849920 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:08.850040 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:08.850043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:08.850082 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:08.850144 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:08.850408 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.851012 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.851827 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.852035 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.852109 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.852312 (+ 203us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.852371 (+ 59us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.852431 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:08.852547 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 18:59:08.852586 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:59:08.852771 7965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.849681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:33651 (local address 127.27.147.130:37403)
0810 18:59:08.849831 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.849848 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:08.849930 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:08.850180 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:08.850183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:08.850193 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 18:59:08.850300 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.850305 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.851126 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.851130 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.851727 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:08.851736 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:08.852128 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:08.852130 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:08.852191 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:08.852218 (+ 27us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 18:59:08.852452 (+ 234us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 18:59:08.852457 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 18:59:08.852541 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 18:59:08.852632 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260810 18:59:08.869601 7125 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.869763 7125 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59478:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:08.871640 7125 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b108f3b0aac34bdcabbdb5d67165c880: Sending DeleteTablet for 3 replicas of tablet a0734ff4b6ed49f3af236f65e5dbdeb0
I20260810 18:59:08.872161 7328 tablet_service.cc:1514] Processing DeleteTablet for tablet a0734ff4b6ed49f3af236f65e5dbdeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:51230
I20260810 18:59:08.872294 7540 tablet_service.cc:1514] Processing DeleteTablet for tablet a0734ff4b6ed49f3af236f65e5dbdeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:55122
I20260810 18:59:08.872345 7917 tablet_replica.cc:330] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e: stopping tablet replica
I20260810 18:59:08.872337 7749 tablet_service.cc:1514] Processing DeleteTablet for tablet a0734ff4b6ed49f3af236f65e5dbdeb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:08 UTC) from {username='oryx', principal='oryx/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:41146
I20260810 18:59:08.872457 7914 tablet_replica.cc:330] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: stopping tablet replica
I20260810 18:59:08.872498 7915 tablet_replica.cc:330] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: stopping tablet replica
I20260810 18:59:08.872485 7917 raft_consensus.cc:2229] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:59:08.872592 7915 raft_consensus.cc:2229] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.872591 7914 raft_consensus.cc:2229] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:08.872666 7917 raft_consensus.cc:2258] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.872699 7914 raft_consensus.cc:2258] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.872701 7915 raft_consensus.cc:2258] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:08.873034 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 7230
I20260810 18:59:08.873468 7917 ts_tablet_manager.cc:1905] T a0734ff4b6ed49f3af236f65e5dbdeb0 P aee157df2c2045158ead0ec9b2f7ea7e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.879546 7915 ts_tablet_manager.cc:1905] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.882371 7914 ts_tablet_manager.cc:1905] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:08.883090 7915 ts_tablet_manager.cc:1918] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:59:08.883178 7915 log.cc:1196] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/a0734ff4b6ed49f3af236f65e5dbdeb0
I20260810 18:59:08.883469 7915 ts_tablet_manager.cc:1939] T a0734ff4b6ed49f3af236f65e5dbdeb0 P b0e22c4a7f264aba99136dfb225630d6: Deleting consensus metadata
I20260810 18:59:08.884276 7103 catalog_manager.cc:4841] TS b0e22c4a7f264aba99136dfb225630d6 (127.27.147.131:38751): tablet a0734ff4b6ed49f3af236f65e5dbdeb0 (table test-table [id=67aa02ffa5ed4faf8dd65a10c4074769]) successfully deleted
I20260810 18:59:08.884718 7914 ts_tablet_manager.cc:1918] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 18:59:08.884815 7914 log.cc:1196] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/a0734ff4b6ed49f3af236f65e5dbdeb0
I20260810 18:59:08.885152 7914 ts_tablet_manager.cc:1939] T a0734ff4b6ed49f3af236f65e5dbdeb0 P 09050963869d44b6a9e7268448139048: Deleting consensus metadata
I20260810 18:59:08.885813 7103 catalog_manager.cc:4841] TS 09050963869d44b6a9e7268448139048 (127.27.147.130:43639): tablet a0734ff4b6ed49f3af236f65e5dbdeb0 (table test-table [id=67aa02ffa5ed4faf8dd65a10c4074769]) successfully deleted
W20260810 18:59:08.891675 7104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:33651 (error 108)
I20260810 18:59:08.891921 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 7441
I20260810 18:59:08.896212 8014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:08.895718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:33651 (local address 127.0.0.1:51264)
0810 18:59:08.896000 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:08.896089 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:33651: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":99,"threads_started":1}
W20260810 18:59:08.897123 7104 catalog_manager.cc:4584] TS aee157df2c2045158ead0ec9b2f7ea7e (127.27.147.129:33651): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet a0734ff4b6ed49f3af236f65e5dbdeb0: Network error: Client connection negotiation failed: client connection to 127.27.147.129:33651: connect: Connection refused (error 111)
I20260810 18:59:08.906296 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 7648
I20260810 18:59:08.917598 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 7083
2026-08-10T18:59:08Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3523 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[8020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[8020](info): set up 2 sockets
Aug 10 18:59:08 dist-test-slave-znh6 krb5kdc[8020](info): commencing operation
krb5kdc: starting...
W20260810 18:59:10.184124 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:59:10 dist-test-slave-znh6 krb5kdc[8020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:59:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:59:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:59:10.315891 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37317
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34001
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:10.429453 8037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:10.429726 8037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:10.429809 8037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:10.433822 8037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:10.433915 8037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:10.433954 8037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:10.433990 8037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:10.434021 8037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:10.439227 8037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34001
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:10.440233 8037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:10.441291 8037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:10.449919 8047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:10.449899 8048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:10.449903 8051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:10.449977 8037 server_base.cc:1034] running on GCE node
I20260810 18:59:10.450728 8037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:10.451836 8037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:10.452996 8037 hybrid_clock.cc:648] HybridClock initialized: now 1786388350452970 us; error 38 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T18:59:10Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1547 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:59:10 dist-test-slave-znh6 krb5kdc[8057](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:59:10 dist-test-slave-znh6 krb5kdc[8057](info): set up 2 sockets
Aug 10 18:59:10 dist-test-slave-znh6 krb5kdc[8057](info): commencing operation
krb5kdc: starting...
W20260810 18:59:11.739934 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.244s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:59:11 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:59:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:59:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:59:11.865228 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38623
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:11.976174 8078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:11.976493 8078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:11.976579 8078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:11.980566 8078 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:11.980659 8078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:11.980695 8078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:11.980749 8078 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:11.980791 8078 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:11.986014 8078 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8078
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:11.987118 8078 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:11.988181 8078 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:11.996950 8085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:11.997066 8078 server_base.cc:1034] running on GCE node
W20260810 18:59:11.996929 8086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:11.996910 8088 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:11.997740 8078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:11.998867 8078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:12.000084 8078 hybrid_clock.cc:648] HybridClock initialized: now 1786388352000062 us; error 35 us; skew 500 ppm
Aug 10 18:59:11 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388351, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:12.002954 8078 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:59:12.003880 8078 webserver.cc:460] Webserver started at http://127.27.147.190:46275/ using document root <none> and password file <none>
I20260810 18:59:12.004451 8078 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:12.004542 8078 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:12.004781 8078 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:12.006978 8078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "14c169e103d40df44ffb5c82fe47ab58"
server_key_iv: "f18b0d88a69ff840b15921de87253b0e"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.007567 8078 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "14c169e103d40df44ffb5c82fe47ab58"
server_key_iv: "f18b0d88a69ff840b15921de87253b0e"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.011209 8078 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:59:12.013592 8098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.014890 8078 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:12.015038 8078 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "14c169e103d40df44ffb5c82fe47ab58"
server_key_iv: "f18b0d88a69ff840b15921de87253b0e"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.015148 8078 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:12.043223 8078 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:12.046093 8078 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:12.046298 8078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:12.053625 8078 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38623
I20260810 18:59:12.053643 8171 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38623 every 8 connection(s)
I20260810 18:59:12.054549 8078 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:59:12.057356 8172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:12.061721 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 8078
I20260810 18:59:12.061890 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:59:12.062163 28238 external_mini_cluster.cc:1428] Setting key 3eeb43cb29fe27de65d176a8d46d8172
I20260810 18:59:12.063126 8172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Bootstrap starting.
I20260810 18:59:12.065366 8172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:12.066315 8172 log.cc:824] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:12.068543 8172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: No bootstrap required, opened a new log
Aug 10 18:59:12 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388351, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:12.072757 8172 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:12.072989 8172 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:12.073055 8172 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae89de5bba3a46fb8ab89e6e90e8b6d0, State: Initialized, Role: FOLLOWER
I20260810 18:59:12.073556 8172 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:12.073722 8172 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:59:12.073797 8172 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:59:12.073911 8172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:12.074980 8172 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:12.075340 8172 leader_election.cc:302] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae89de5bba3a46fb8ab89e6e90e8b6d0; no voters:
I20260810 18:59:12.075673 8172 leader_election.cc:288] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:59:12.075757 8184 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:12.075999 8184 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 LEADER]: Becoming Leader. State: Replica: ae89de5bba3a46fb8ab89e6e90e8b6d0, State: Running, Role: LEADER
I20260810 18:59:12.076318 8184 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:12.076988 8172 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:12.077627 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:12.063630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59774 (local address 127.27.147.190:38623)
0810 18:59:12.064014 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:12.064039 (+ 25us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:12.064072 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:59:12.064674 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:12.064690 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:12.064993 (+ 303us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:12.065252 (+ 259us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:12.066197 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.067418 (+ 1221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.068584 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.069071 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.069188 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:12.071968 (+ 2780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:12.071974 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:12.071987 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:12.072013 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:12.074017 (+ 2004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.074451 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.074457 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.074463 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.074613 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.074927 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.074932 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.074934 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.075428 (+ 494us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:12.075666 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:12.076306 (+ 640us) server_negotiation.cc:299] Negotiation successful
0810 18:59:12.076682 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":124,"threads_started":1}
I20260810 18:59:12.077622 8184 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae89de5bba3a46fb8ab89e6e90e8b6d0. Latest consensus state: current_term: 1 leader_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } } }
I20260810 18:59:12.077745 8184 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:12.078114 8192 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:59:12.078451 8185 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } } }
I20260810 18:59:12.078620 8185 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:12.081740 8192 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:59:12.087118 8192 catalog_manager.cc:1347] Generated new cluster ID: a5613d1968eb44f998e45066853d8022
I20260810 18:59:12.087196 8192 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:59:12.098380 8192 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:59:12.099335 8192 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:59:12.104789 8192 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Generated new TSK 0
I20260810 18:59:12.105381 8192 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:59:12.137180 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38623
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:12.270740 8211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:12.271056 8211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:12.271135 8211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:12.275211 8211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:12.275293 8211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:12.275379 8211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:59:12.280854 8211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:12.281946 8211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:12.283017 8211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 18:59:12.292973 8211 server_base.cc:1034] running on GCE node
W20260810 18:59:12.292929 8217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:12.292936 8216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:12.292929 8219 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:12.293597 8211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:12.294164 8211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:12.295364 8211 hybrid_clock.cc:648] HybridClock initialized: now 1786388352295337 us; error 37 us; skew 500 ppm
Aug 10 18:59:12 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:12.298070 8211 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:59:12.298947 8211 webserver.cc:460] Webserver started at http://127.27.147.129:37401/ using document root <none> and password file <none>
I20260810 18:59:12.299499 8211 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:12.299562 8211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:12.299810 8211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:12.301811 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "f1eb8cc716744ea4b54ac0ae32e4e326"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "eaef85131138c96350d36d15971ecf70"
server_key_iv: "438f5bf111b3d5bc961731ccf053cb39"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.302379 8211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "f1eb8cc716744ea4b54ac0ae32e4e326"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "eaef85131138c96350d36d15971ecf70"
server_key_iv: "438f5bf111b3d5bc961731ccf053cb39"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.306123 8211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 18:59:12.308524 8227 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.309645 8211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:59:12.309768 8211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "f1eb8cc716744ea4b54ac0ae32e4e326"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "eaef85131138c96350d36d15971ecf70"
server_key_iv: "438f5bf111b3d5bc961731ccf053cb39"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.309863 8211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:12.324770 8211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:12.327976 8211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:12.328177 8211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:12.328770 8211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:12.329583 8211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:12.329633 8211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.329672 8211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:12.329721 8211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.338382 8211 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:42297
I20260810 18:59:12.338447 8419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:42297 every 8 connection(s)
I20260810 18:59:12.339303 8211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:59:12.344293 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 8211
I20260810 18:59:12.344452 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:59:12.344726 28238 external_mini_cluster.cc:1428] Setting key c0c5af393b12e3497af9473fbd34e55a
Aug 10 18:59:12 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:12.353760 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:12.341174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:42251 (local address 127.27.147.190:38623)
0810 18:59:12.341328 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:12.341333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:12.342201 (+ 868us) server_negotiation.cc:407] Connection header received
0810 18:59:12.343607 (+ 1406us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:12.343610 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:12.343658 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:12.343723 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:12.344982 (+ 1259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.345588 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.346972 (+ 1384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.347402 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.347524 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:12.349973 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:12.349977 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:12.349979 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:12.350002 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:12.351518 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.352026 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.352031 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.352033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.352115 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.352530 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.352535 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.352537 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.352748 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:12.352846 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:12.353536 (+ 690us) server_negotiation.cc:299] Negotiation successful
0810 18:59:12.353637 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 18:59:12.355015 8424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:12.341600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:42251)
0810 18:59:12.342050 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:12.342091 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:12.343383 (+ 1292us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:12.343890 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:12.343897 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:12.344240 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:12.344821 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:12.344842 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.345713 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.345716 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:12.346786 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:12.346796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.347552 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.347554 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:12.347695 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:12.348345 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:12.348370 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:12.349799 (+ 1429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:12.351639 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:12.351644 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:12.351659 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:12.351920 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:12.352208 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:12.352212 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:12.352214 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:12.352361 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:12.352876 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:12.352886 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:12.353238 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 18:59:12.353490 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 18:59:12.353846 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 18:59:12.356302 8421 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38623
I20260810 18:59:12.356519 8421 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:12.357018 8421 heartbeater.cc:511] Master 127.27.147.190:38623 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 18:59:12.358718 8117 ts_manager.cc:194] Registered new tserver with Master: f1eb8cc716744ea4b54ac0ae32e4e326 (127.27.147.129:42297)
I20260810 18:59:12.359891 8117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:42251
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:59:12.375569 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38623
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:12.492383 8427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:12.492641 8427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:12.492725 8427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:12.496681 8427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:12.496758 8427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:12.496845 8427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:59:12.501991 8427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:12.503026 8427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:12.504037 8427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:12.513864 8439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:12.513864 8433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:12.514154 8427 server_base.cc:1034] running on GCE node
W20260810 18:59:12.514318 8434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:12.514864 8427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:12.515642 8427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:12.516875 8427 hybrid_clock.cc:648] HybridClock initialized: now 1786388352516856 us; error 34 us; skew 500 ppm
Aug 10 18:59:12 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:12.519768 8427 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:59:12.520666 8427 webserver.cc:460] Webserver started at http://127.27.147.130:34085/ using document root <none> and password file <none>
I20260810 18:59:12.521215 8427 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:12.521308 8427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:12.521543 8427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:12.523698 8427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "bdd87a14320d463c8cfddfeacaf65208"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "76d8e550769ead916e1ef867bc771d52"
server_key_iv: "e7a6970f3d71c7c6a8aef87b2926175d"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.524256 8427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "bdd87a14320d463c8cfddfeacaf65208"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "76d8e550769ead916e1ef867bc771d52"
server_key_iv: "e7a6970f3d71c7c6a8aef87b2926175d"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.527935 8427 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:59:12.530483 8447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.531769 8427 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:12.531901 8427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "bdd87a14320d463c8cfddfeacaf65208"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "76d8e550769ead916e1ef867bc771d52"
server_key_iv: "e7a6970f3d71c7c6a8aef87b2926175d"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.532004 8427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:12.551589 8427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:12.554934 8427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:12.555135 8427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:12.555743 8427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:12.556568 8427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:12.556617 8427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.556687 8427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:12.556726 8427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.565546 8427 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:43187
I20260810 18:59:12.565577 8633 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:43187 every 8 connection(s)
I20260810 18:59:12.566404 8427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:59:12.572896 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 8427
I20260810 18:59:12.573035 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:59:12.573345 28238 external_mini_cluster.cc:1428] Setting key 5cf2cf7a5cb487bb4434d24d965d3778
Aug 10 18:59:12 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:12.580816 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:12.568280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:51975 (local address 127.27.147.190:38623)
0810 18:59:12.568422 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:12.568425 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:12.569231 (+ 806us) server_negotiation.cc:407] Connection header received
0810 18:59:12.570605 (+ 1374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:12.570608 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:12.570662 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:12.570771 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:12.571925 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.572465 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.573766 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.574003 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.574117 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:12.576584 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:12.576590 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:12.576596 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:12.576631 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:12.578310 (+ 1679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.578876 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.578879 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.578881 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.578990 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.579392 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.579395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.579396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.579573 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:12.579714 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:12.580383 (+ 669us) server_negotiation.cc:299] Negotiation successful
0810 18:59:12.580568 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:59:12.581634 8638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:12.568627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:51975)
0810 18:59:12.569088 (+ 461us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:12.569128 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:12.570399 (+ 1271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:12.570976 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:12.570983 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:12.571317 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:12.571765 (+ 448us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:12.571778 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.572591 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.572594 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:12.573631 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:12.573642 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.574155 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.574160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:12.574323 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:12.574921 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:12.574937 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:12.576403 (+ 1466us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:12.578442 (+ 2039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:12.578447 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:12.578462 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:12.578726 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:12.579084 (+ 358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:12.579091 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:12.579095 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:12.579287 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:12.579682 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:12.579692 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:12.580041 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 18:59:12.580293 (+ 252us) client_negotiation.cc:240] Negotiation successful
0810 18:59:12.580665 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":130,"threads_started":1}
I20260810 18:59:12.582780 8634 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38623
I20260810 18:59:12.583076 8634 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:12.583710 8634 heartbeater.cc:511] Master 127.27.147.190:38623 requested a full tablet report, sending...
I20260810 18:59:12.584836 8117 ts_manager.cc:194] Registered new tserver with Master: bdd87a14320d463c8cfddfeacaf65208 (127.27.147.130:43187)
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:59:12.585450 8117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:51975
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:59:12.603410 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38623
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:12.714437 8643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:12.714695 8643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:12.714774 8643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:12.718696 8643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:12.718782 8643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:12.718899 8643 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:12.724002 8643 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:12.724949 8643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:12.725920 8643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:12.735575 8653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:12.735605 8651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:12.735683 8650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:12.735708 8643 server_base.cc:1034] running on GCE node
I20260810 18:59:12.736310 8643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:12.736873 8643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:12.738054 8643 hybrid_clock.cc:648] HybridClock initialized: now 1786388352738038 us; error 25 us; skew 500 ppm
Aug 10 18:59:12 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:12.740772 8643 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:59:12.741596 8643 webserver.cc:460] Webserver started at http://127.27.147.131:34287/ using document root <none> and password file <none>
I20260810 18:59:12.742172 8643 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:12.742265 8643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:12.742498 8643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:12.744558 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "254307f78cd442d0b89ae19fbd23f3d5"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "7a58ce2d10d04541d777c6ec595efabd"
server_key_iv: "2b4a94e3fcf273f464c53f9fa232d8f9"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.745129 8643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "254307f78cd442d0b89ae19fbd23f3d5"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "7a58ce2d10d04541d777c6ec595efabd"
server_key_iv: "2b4a94e3fcf273f464c53f9fa232d8f9"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.748831 8643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:59:12.751266 8662 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.752437 8643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:12.752575 8643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "254307f78cd442d0b89ae19fbd23f3d5"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "7a58ce2d10d04541d777c6ec595efabd"
server_key_iv: "2b4a94e3fcf273f464c53f9fa232d8f9"
server_key_version: "encryptionkey@0"
I20260810 18:59:12.752683 8643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:12.763026 8643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:12.765631 8643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:12.765836 8643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:12.766393 8643 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:12.767246 8643 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:12.767328 8643 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.767392 8643 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:12.767441 8643 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:12.775998 8643 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:43039
I20260810 18:59:12.776070 8838 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:43039 every 8 connection(s)
I20260810 18:59:12.776908 8643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:12.779752 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 8643
I20260810 18:59:12.779911 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:59:12.780220 28238 external_mini_cluster.cc:1428] Setting key 5072e4073afa6f6bfd5decc67374d097
Aug 10 18:59:12 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388352, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:12.790417 8178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:12.778932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:51069 (local address 127.27.147.190:38623)
0810 18:59:12.779080 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:12.779085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:12.779973 (+ 888us) server_negotiation.cc:407] Connection header received
0810 18:59:12.781533 (+ 1560us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:12.781537 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:12.781599 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:12.781674 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:12.783027 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.783605 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.784686 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.784897 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.784972 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:12.787401 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:12.787404 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:12.787406 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:12.787429 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:12.788646 (+ 1217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.789101 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.789103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.789104 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.789185 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:12.789500 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:12.789502 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:12.789503 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:12.789672 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:12.789754 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:12.790188 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 18:59:12.790284 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:59:12.791245 8842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:12.779262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:51069)
0810 18:59:12.779791 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:12.779853 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:12.781307 (+ 1454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:12.781878 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:12.781885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:12.782221 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:12.782893 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:12.782909 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.783723 (+ 814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.783726 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:12.784582 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:12.784591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:12.785026 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:12.785029 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:12.785166 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:12.785756 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:12.785778 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:12.787241 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:12.788749 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:12.788753 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:12.788766 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:12.789006 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:12.789269 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:12.789271 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:12.789272 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:12.789399 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:12.789766 (+ 367us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:12.789770 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:12.789982 (+ 212us) client_negotiation.cc:769] Sending connection context
0810 18:59:12.790123 (+ 141us) client_negotiation.cc:240] Negotiation successful
0810 18:59:12.790379 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":107,"threads_started":1}
I20260810 18:59:12.792405 8840 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38623
I20260810 18:59:12.792665 8840 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:12.793267 8840 heartbeater.cc:511] Master 127.27.147.190:38623 requested a full tablet report, sending...
I20260810 18:59:12.794361 8117 ts_manager.cc:194] Registered new tserver with Master: 254307f78cd442d0b89ae19fbd23f3d5 (127.27.147.131:43039)
I20260810 18:59:12.794968 8117 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:51069
I20260810 18:59:12.805864 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab.
Entry for principal oryx/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab.
WARNING: no policy specified for oryx/127.27.147.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab.
Entry for principal oryx/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab.
WARNING: no policy specified for oryx/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab.
Entry for principal oryx/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab.
I20260810 18:59:12.881868 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 8211
I20260810 18:59:12.903916 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 8427
I20260810 18:59:12.913820 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 8643
I20260810 18:59:12.923619 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 8078
I20260810 18:59:12.932206 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38623
--webserver_interface=127.27.147.190
--webserver_port=46275
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:13.045528 8850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:13.045825 8850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:13.045907 8850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:13.049894 8850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:13.049979 8850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:13.050019 8850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:13.050055 8850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:13.050091 8850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:13.055366 8850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=46275
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8850
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:13.056371 8850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:13.057368 8850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:13.066000 8859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:13.066006 8860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:13.066025 8850 server_base.cc:1034] running on GCE node
W20260810 18:59:13.066061 8863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:13.066856 8850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:13.068010 8850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:13.069205 8850 hybrid_clock.cc:648] HybridClock initialized: now 1786388353069188 us; error 26 us; skew 500 ppm
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388353, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:13.072010 8850 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:59:13.072885 8850 webserver.cc:460] Webserver started at http://127.27.147.190:46275/ using document root <none> and password file <none>
I20260810 18:59:13.073446 8850 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:13.073563 8850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:13.077330 8850 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:13.079638 8875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.080811 8850 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:13.080963 8850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "14c169e103d40df44ffb5c82fe47ab58"
server_key_iv: "f18b0d88a69ff840b15921de87253b0e"
server_key_version: "encryptionkey@0"
I20260810 18:59:13.081496 8850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:13.105955 8850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:13.108839 8850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:13.109040 8850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:13.116321 8850 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38623
I20260810 18:59:13.116335 8959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38623 every 8 connection(s)
I20260810 18:59:13.117362 8850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:59:13.118700 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 8850
I20260810 18:59:13.119172 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:42297
--local_ip_for_outbound_sockets=127.27.147.129
--tserver_master_addrs=127.27.147.190:38623
--webserver_port=37401
--webserver_interface=127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 18:59:13.122254 8960 sys_catalog.cc:263] Verifying existing consensus state
I20260810 18:59:13.124032 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Bootstrap starting.
I20260810 18:59:13.141044 8960 log.cc:824] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:13.147792 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 18:59:13.148190 8960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Bootstrap complete.
I20260810 18:59:13.152575 8960 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:13.152981 8960 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae89de5bba3a46fb8ab89e6e90e8b6d0, State: Initialized, Role: FOLLOWER
I20260810 18:59:13.153482 8960 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:13.153597 8960 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:59:13.153687 8960 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:59:13.153817 8960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 1 FOLLOWER]: Advancing to term 2
I20260810 18:59:13.155840 8960 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:13.156180 8960 leader_election.cc:302] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae89de5bba3a46fb8ab89e6e90e8b6d0; no voters:
I20260810 18:59:13.156522 8960 leader_election.cc:288] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 18:59:13.156612 8966 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 18:59:13.156880 8966 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [term 2 LEADER]: Becoming Leader. State: Replica: ae89de5bba3a46fb8ab89e6e90e8b6d0, State: Running, Role: LEADER
I20260810 18:59:13.157232 8966 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } }
I20260810 18:59:13.157770 8960 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:13.158239 8966 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae89de5bba3a46fb8ab89e6e90e8b6d0. Latest consensus state: current_term: 2 leader_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } } }
I20260810 18:59:13.158361 8966 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:13.158766 8969 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae89de5bba3a46fb8ab89e6e90e8b6d0" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38623 } } }
I20260810 18:59:13.158954 8969 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:13.159003 8972 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:59:13.162369 8972 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:59:13.163157 8972 catalog_manager.cc:1259] Loaded cluster ID: a5613d1968eb44f998e45066853d8022
I20260810 18:59:13.163239 8972 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:59:13.164937 8972 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:59:13.165531 8972 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ae89de5bba3a46fb8ab89e6e90e8b6d0: Loaded TSK: 0
I20260810 18:59:13.166474 8972 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 18:59:13.244478 8962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:13.244765 8962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:13.244858 8962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:13.248970 8962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:13.249053 8962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:13.249143 8962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:59:13.254333 8962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.129.keytab
--principal=oryx/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:42297
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=37401
--tserver_master_addrs=127.27.147.190:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:13.255451 8962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:13.256518 8962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:13.267663 8998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:13.267661 8995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:13.267683 8996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:13.268075 8962 server_base.cc:1034] running on GCE node
I20260810 18:59:13.268581 8962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:13.269268 8962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:13.270463 8962 hybrid_clock.cc:648] HybridClock initialized: now 1786388353270431 us; error 51 us; skew 500 ppm
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388353, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:13.273517 8962 init.cc:377] Logged in from keytab as oryx/127.27.147.129@KRBTEST.COM (short username oryx)
I20260810 18:59:13.274605 8962 webserver.cc:460] Webserver started at http://127.27.147.129:37401/ using document root <none> and password file <none>
I20260810 18:59:13.275357 8962 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:13.275472 8962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:13.280589 8962 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:59:13.283209 9007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.284730 8962 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 18:59:13.284863 8962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "f1eb8cc716744ea4b54ac0ae32e4e326"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "eaef85131138c96350d36d15971ecf70"
server_key_iv: "438f5bf111b3d5bc961731ccf053cb39"
server_key_version: "encryptionkey@0"
I20260810 18:59:13.285528 8962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:13.305567 8962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:13.308893 8962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:13.309109 8962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:13.309753 8962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:13.310590 8962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:13.310639 8962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.310702 8962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:13.310738 8962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.319720 8962 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:42297
I20260810 18:59:13.319763 9182 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:42297 every 8 connection(s)
I20260810 18:59:13.320761 8962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:59:13.327145 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 8962
I20260810 18:59:13.327590 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:43187
--local_ip_for_outbound_sockets=127.27.147.130
--tserver_master_addrs=127.27.147.190:38623
--webserver_port=34085
--webserver_interface=127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:59:13.334462 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.322848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:50841 (local address 127.27.147.190:38623)
0810 18:59:13.323394 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.323414 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.323609 (+ 195us) server_negotiation.cc:407] Connection header received
0810 18:59:13.325150 (+ 1541us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.325160 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.325442 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.325761 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.326917 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.328054 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.329198 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.329657 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.329847 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.333787 (+ 3940us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:50841: BlockingRecv error: recv got EOF from 127.27.147.129:50841 (error 108)
Metrics: {"server-negotiator.queue_time_us":340,"thread_start_us":126,"threads_started":1}
I20260810 18:59:13.334619 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.322965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:50841)
0810 18:59:13.323446 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.323504 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.324747 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.325844 (+ 1097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.325853 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.326269 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.326712 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.326726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.328191 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.328194 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.329067 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.329077 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.329835 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.329840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.330044 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.330653 (+ 609us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.330669 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.333609 (+ 2940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":281,"thread_start_us":139,"threads_started":1}
W20260810 18:59:13.335255 9183 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38623 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.147.190:38623: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:13.340951 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.335725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:56223)
0810 18:59:13.335820 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.335833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.335938 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.336193 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.336196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.336358 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.336445 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.336450 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.337164 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.337171 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.337717 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.337725 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.338174 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.338177 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.338269 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.338757 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.338769 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.340796 (+ 2027us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":29}
W20260810 18:59:13.341038 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.335842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:56223 (local address 127.27.147.190:38623)
0810 18:59:13.335996 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.336002 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.336015 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:13.336055 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.336057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.336093 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.336178 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.336564 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.337057 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.337841 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.338062 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.338172 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.340918 (+ 2746us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:56223: BlockingRecv error: recv got EOF from 127.27.147.129:56223 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:13.346400 9188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.341448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:43867)
0810 18:59:13.341539 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.341553 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.341624 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.341839 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.341841 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.341959 (+ 118us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.342026 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.342030 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.343040 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.343042 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.343566 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.343574 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.344124 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.344126 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.344182 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.344636 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.344648 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.346314 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 18:59:13.346593 9183 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:59:13.346640 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.341517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:43867 (local address 127.27.147.190:38623)
0810 18:59:13.341620 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.341624 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.341638 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:13.341678 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.341681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.341721 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.341817 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.342137 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.342937 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.343672 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.344019 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.344118 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.346497 (+ 2379us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:43867: BlockingRecv error: recv got EOF from 127.27.147.129:43867 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 18:59:13.445698 9189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:13.445935 9189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:13.445974 9189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:13.449873 9189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:13.449923 9189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:13.449978 9189 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:59:13.454996 9189 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.130.keytab
--principal=oryx/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:43187
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=34085
--tserver_master_addrs=127.27.147.190:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:13.455996 9189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:13.456923 9189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:13.467001 9197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:13.466957 9194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:13.466957 9205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:13.467330 9189 server_base.cc:1034] running on GCE node
I20260810 18:59:13.467777 9189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:13.468410 9189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:13.469601 9189 hybrid_clock.cc:648] HybridClock initialized: now 1786388353469580 us; error 32 us; skew 500 ppm
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388353, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:13.472373 9189 init.cc:377] Logged in from keytab as oryx/127.27.147.130@KRBTEST.COM (short username oryx)
I20260810 18:59:13.473156 9189 webserver.cc:460] Webserver started at http://127.27.147.130:34085/ using document root <none> and password file <none>
I20260810 18:59:13.473690 9189 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:13.473762 9189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:13.477442 9189 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:13.479453 9214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.480633 9189 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:13.480769 9189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "bdd87a14320d463c8cfddfeacaf65208"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "76d8e550769ead916e1ef867bc771d52"
server_key_iv: "e7a6970f3d71c7c6a8aef87b2926175d"
server_key_version: "encryptionkey@0"
I20260810 18:59:13.481276 9189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:13.500658 9189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:13.503510 9189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:13.503711 9189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:13.504289 9189 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:13.505112 9189 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:13.505160 9189 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.505230 9189 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:13.505271 9189 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.514362 9189 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:43187
I20260810 18:59:13.514386 9400 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:43187 every 8 connection(s)
I20260810 18:59:13.515430 9189 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:59:13.525581 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 9189
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:13.526161 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:43039
--local_ip_for_outbound_sockets=127.27.147.131
--tserver_master_addrs=127.27.147.190:38623
--webserver_port=34287
--webserver_interface=127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34189
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:59:13.527966 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.517506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:42029 (local address 127.27.147.190:38623)
0810 18:59:13.517770 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.517775 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.519225 (+ 1450us) server_negotiation.cc:407] Connection header received
0810 18:59:13.520858 (+ 1633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.520861 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.520918 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.520994 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.522238 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.522780 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.523937 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.524158 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.524241 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.527784 (+ 3543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:42029: BlockingRecv error: recv got EOF from 127.27.147.130:42029 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
I20260810 18:59:13.528529 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.517979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:42029)
0810 18:59:13.519000 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.519059 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.520644 (+ 1585us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.521169 (+ 525us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.521176 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.521526 (+ 350us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.522100 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.522114 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.522927 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.522930 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.523820 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.523830 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.524294 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.524296 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.524439 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.525016 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.525033 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.527588 (+ 2555us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":129,"threads_started":1}
W20260810 18:59:13.529013 9401 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38623 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.147.190:38623: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:13.534868 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.529441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:49289)
0810 18:59:13.529701 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.529715 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.529804 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.530072 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.530075 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.530242 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.530327 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.530332 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.531117 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.531122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.531758 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.531767 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.532209 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.532213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.532308 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.532797 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.532819 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.534704 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197}
W20260810 18:59:13.534950 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.529577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:49289 (local address 127.27.147.190:38623)
0810 18:59:13.529701 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.529707 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.529762 (+ 55us) server_negotiation.cc:407] Connection header received
0810 18:59:13.529931 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.529933 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.529974 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.530058 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.530458 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.531012 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.531881 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.532098 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.532208 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.534836 (+ 2628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:49289: BlockingRecv error: recv got EOF from 127.27.147.130:49289 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:13.540158 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.535449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:46107)
0810 18:59:13.535537 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.535552 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.535646 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.535902 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.535905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.536061 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.536123 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.536128 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.536811 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.536816 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.537324 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.537332 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.537765 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.537768 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.537843 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.538286 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.538307 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.540030 (+ 1723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":27}
W20260810 18:59:13.540243 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.535572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:46107 (local address 127.27.147.190:38623)
0810 18:59:13.535720 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.535724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.535743 (+ 19us) server_negotiation.cc:407] Connection header received
0810 18:59:13.535772 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.535774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.535805 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.535887 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.536231 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.536706 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.537448 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.537656 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.537763 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.540134 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:46107: BlockingRecv error: recv got EOF from 127.27.147.130:46107 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260810 18:59:13.540401 9401 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 18:59:13.641134 9404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:13.641369 9404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:13.641410 9404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:13.645538 9404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:13.645596 9404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:13.645690 9404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:13.650760 9404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34189
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/krb5kdc/oryx_127.27.147.131.keytab
--principal=oryx/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:43039
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=34287
--tserver_master_addrs=127.27.147.190:38623
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:13.651782 9404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:13.652746 9404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:13.662715 9415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:13.662957 9404 server_base.cc:1034] running on GCE node
W20260810 18:59:13.663002 9412 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:13.663273 9417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:13.663688 9404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:13.664263 9404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:13.665421 9404 hybrid_clock.cc:648] HybridClock initialized: now 1786388353665404 us; error 29 us; skew 500 ppm
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388353, etypes {rep=17 tkt=17 ses=17}, oryx/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:13.668231 9404 init.cc:377] Logged in from keytab as oryx/127.27.147.131@KRBTEST.COM (short username oryx)
I20260810 18:59:13.669052 9404 webserver.cc:460] Webserver started at http://127.27.147.131:34287/ using document root <none> and password file <none>
I20260810 18:59:13.669597 9404 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:13.669694 9404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:13.673437 9404 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:13.675487 9426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.676657 9404 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:13.676776 9404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "254307f78cd442d0b89ae19fbd23f3d5"
format_stamp: "Formatted at 2026-08-10 18:59:12 on dist-test-slave-znh6"
server_key: "7a58ce2d10d04541d777c6ec595efabd"
server_key_iv: "2b4a94e3fcf273f464c53f9fa232d8f9"
server_key_version: "encryptionkey@0"
I20260810 18:59:13.677279 9404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:13.704021 9404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:13.707208 9404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:13.707417 9404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:13.708015 9404 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:13.708835 9404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:13.708882 9404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.708972 9404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:13.709012 9404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:13.717662 9404 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:43039
I20260810 18:59:13.717701 9598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:43039 every 8 connection(s)
I20260810 18:59:13.718633 9404 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:13.724233 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 9404
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
W20260810 18:59:13.729954 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.720331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:58783 (local address 127.27.147.190:38623)
0810 18:59:13.720479 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.720483 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.721225 (+ 742us) server_negotiation.cc:407] Connection header received
0810 18:59:13.722504 (+ 1279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.722506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.722556 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.722628 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.723810 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.724560 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.726121 (+ 1561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.726473 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.726553 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.729833 (+ 3280us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:58783: BlockingRecv error: recv got EOF from 127.27.147.131:58783 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 18:59:13.730656 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.720634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:58783)
0810 18:59:13.721066 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.721108 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.722309 (+ 1201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.722797 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.722804 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.723207 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.723685 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.723698 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.724715 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.724722 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.725996 (+ 1274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.726005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.726626 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.726630 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.726834 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.727422 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.727438 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.729732 (+ 2294us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":109,"threads_started":1}
W20260810 18:59:13.731123 9599 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38623 (0 consecutive failures): Not authorized: Failed to ping master at 127.27.147.190:38623: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
I20260810 18:59:13.733861 9605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.725200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59786 (local address 127.27.147.190:38623)
0810 18:59:13.725475 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.725480 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.725494 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:13.725530 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.725533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.725573 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.725662 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.726430 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.727177 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.728071 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.728297 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.728418 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.729765 (+ 1347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:13.729771 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:13.729784 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:13.729811 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:13.731874 (+ 2063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:13.732314 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:13.732319 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:13.732323 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:13.732441 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:13.732730 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:13.732734 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:13.732736 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:13.733050 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:13.733248 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:13.733531 (+ 283us) server_negotiation.cc:299] Negotiation successful
0810 18:59:13.733747 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":190,"thread_start_us":88,"threads_started":1}
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:13.736493 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.731552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:60353)
0810 18:59:13.731658 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.731674 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.731751 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.732017 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.732020 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.732162 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.732250 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.732255 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.732940 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.732945 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.733462 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.733471 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.733985 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.733988 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.734083 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.734523 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.734535 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.736376 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":36}
W20260810 18:59:13.736649 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.731633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:60353 (local address 127.27.147.190:38623)
0810 18:59:13.731737 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.731741 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.731753 (+ 12us) server_negotiation.cc:407] Connection header received
0810 18:59:13.731866 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.731869 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.731914 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.731995 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.732363 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.732831 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.733584 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.733858 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.733963 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.736520 (+ 2557us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:60353: BlockingRecv error: recv got EOF from 127.27.147.131:60353 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:13 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:13.741083 9602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:13.737030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:58837)
0810 18:59:13.737138 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:13.737151 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:13.737222 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:13.737385 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:13.737387 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:13.737510 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:13.737569 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.737573 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.738184 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.738186 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.738671 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:13.738678 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.739123 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.739126 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:13.739182 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.739519 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:13.739529 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:13.741007 (+ 1478us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 18:59:13.741259 9186 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:13.737109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:58837 (local address 127.27.147.190:38623)
0810 18:59:13.737242 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:13.737251 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:13.737262 (+ 11us) server_negotiation.cc:407] Connection header received
0810 18:59:13.737289 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:13.737291 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:13.737320 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:13.737379 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:13.737658 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.738094 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.738766 (+ 672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:13.739030 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:13.739105 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:13.741137 (+ 2032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:58837: BlockingRecv error: recv got EOF from 127.27.147.131:58837 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
W20260810 18:59:13.741281 9599 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 18:59:14 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:14 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:14.353039 9606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:14.347273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:45711)
0810 18:59:14.347591 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:14.347605 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:14.347700 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:14.348034 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:14.348038 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:14.348283 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:14.348455 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:14.348467 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.349505 (+ 1038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.349508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:14.350088 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:14.350095 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.350525 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.350527 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:14.350590 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:14.351105 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:14.351116 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:14.352913 (+ 1797us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":91,"threads_started":1}
W20260810 18:59:14.353255 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:14.347422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:45711 (local address 127.27.147.190:38623)
0810 18:59:14.347734 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:14.347739 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:14.347755 (+ 16us) server_negotiation.cc:407] Connection header received
0810 18:59:14.347800 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:14.347803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:14.347871 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:14.347952 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:14.348608 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.349401 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.350199 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.350422 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.350507 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:14.353057 (+ 2550us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:45711: BlockingRecv error: recv got EOF from 127.27.147.129:45711 (error 108)
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":82,"threads_started":1}
Aug 10 18:59:14 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:14 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:14.546489 9610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:14.540985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:58755)
0810 18:59:14.541263 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:14.541277 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:14.541379 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:14.541605 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:14.541607 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:14.541796 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:14.541893 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:14.541898 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.542646 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.542650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:14.543346 (+ 696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:14.543356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.543916 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.543921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:14.544006 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:14.544539 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:14.544551 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:14.546369 (+ 1818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":70,"threads_started":1}
W20260810 18:59:14.546660 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:14.541070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:58755 (local address 127.27.147.190:38623)
0810 18:59:14.541248 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:14.541251 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:14.541348 (+ 97us) server_negotiation.cc:407] Connection header received
0810 18:59:14.541455 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:14.541458 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:14.541501 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:14.541569 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:14.541991 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.542524 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.543479 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.543792 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.543912 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:14.546529 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:58755: BlockingRecv error: recv got EOF from 127.27.147.130:58755 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 18:59:14 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:14 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:14.747380 9611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:14.741803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:53979)
0810 18:59:14.742137 (+ 334us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:14.742158 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:14.742290 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:14.742579 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:14.742582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:14.742790 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:14.742934 (+ 144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:14.742941 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.743728 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.743733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:14.744324 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:14.744333 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.744811 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.744815 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:14.744910 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:14.745409 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:14.745421 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:14.747206 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":83,"threads_started":1}
W20260810 18:59:14.747445 9608 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:14.741922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:53979 (local address 127.27.147.190:38623)
0810 18:59:14.742092 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:14.742098 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:14.742238 (+ 140us) server_negotiation.cc:407] Connection header received
0810 18:59:14.742397 (+ 159us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:14.742401 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:14.742454 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:14.742551 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:14.743067 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.743613 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.744452 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:14.744685 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:14.744795 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:14.747338 (+ 2543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:53979: BlockingRecv error: recv got EOF from 127.27.147.131:53979 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 18:59:15 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:15 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:15.359747 9613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:15.353865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:54925)
0810 18:59:15.354160 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:15.354174 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:15.354264 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:15.354534 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:15.354536 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:15.354718 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:15.354847 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:15.354852 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.355809 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.355814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:15.356616 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:15.356627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.357072 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.357074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:15.357135 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:15.357599 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:15.357610 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:15.359634 (+ 2024us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":113,"threads_started":1}
W20260810 18:59:15.359987 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:15.353954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:54925 (local address 127.27.147.190:38623)
0810 18:59:15.354199 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:15.354203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:15.354218 (+ 15us) server_negotiation.cc:407] Connection header received
0810 18:59:15.354365 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:15.354367 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:15.354411 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:15.354492 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:15.354959 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.355687 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.356726 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.356978 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.357060 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:15.359839 (+ 2779us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:54925: BlockingRecv error: recv got EOF from 127.27.147.129:54925 (error 108)
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":85,"threads_started":1}
Aug 10 18:59:15 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:15 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:15.556901 9616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:15.547593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:55231)
0810 18:59:15.547885 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:15.547897 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:15.548006 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:15.548236 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:15.548238 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:15.548413 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:15.548509 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:15.548513 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.549275 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.549277 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:15.549942 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:15.549952 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.550500 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.550504 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:15.550581 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:15.554664 (+ 4083us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:15.554676 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:15.556762 (+ 2086us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":219,"thread_start_us":116,"threads_started":1}
W20260810 18:59:15.557022 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:15.547657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:55231 (local address 127.27.147.190:38623)
0810 18:59:15.547805 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:15.547810 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:15.547973 (+ 163us) server_negotiation.cc:407] Connection header received
0810 18:59:15.548082 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:15.548085 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:15.548130 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:15.548197 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:15.548616 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.549170 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.550072 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.550378 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.550506 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:15.556904 (+ 6398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:55231: BlockingRecv error: recv got EOF from 127.27.147.130:55231 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Aug 10 18:59:15 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:15 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:15.753517 9618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:15.748298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:36649)
0810 18:59:15.748606 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:15.748621 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:15.748729 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:15.748972 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:15.748975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:15.749155 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:15.749253 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:15.749258 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.749996 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.749999 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:15.750557 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:15.750565 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.750989 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.750992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:15.751056 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:15.751527 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:15.751538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:15.753379 (+ 1841us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":129,"threads_started":1}
W20260810 18:59:15.753654 9615 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:15.748432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:36649 (local address 127.27.147.190:38623)
0810 18:59:15.748584 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:15.748588 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:15.748703 (+ 115us) server_negotiation.cc:407] Connection header received
0810 18:59:15.748809 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:15.748811 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:15.748858 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:15.748931 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:15.749357 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.749901 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.750659 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:15.750895 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:15.750969 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:15.753531 (+ 2562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:36649: BlockingRecv error: recv got EOF from 127.27.147.131:36649 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Aug 10 18:59:16 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:16 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:16.366355 9619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:16.360557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:40119)
0810 18:59:16.360850 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:16.360863 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:16.360956 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:16.361325 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:16.361328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:16.361508 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:16.361607 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:16.361612 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.362401 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.362404 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:16.363040 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:16.363050 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.363654 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.363658 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:16.363744 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:16.364395 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:16.364409 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:16.366194 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":103,"threads_started":1}
W20260810 18:59:16.366474 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:16.360671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:40119 (local address 127.27.147.190:38623)
0810 18:59:16.360927 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:16.360948 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:16.360969 (+ 21us) server_negotiation.cc:407] Connection header received
0810 18:59:16.361096 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:16.361101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:16.361159 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:16.361306 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:16.361733 (+ 427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.362301 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.363162 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.363551 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.363651 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:16.366347 (+ 2696us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:40119: BlockingRecv error: recv got EOF from 127.27.147.129:40119 (error 108)
Metrics: {"server-negotiator.queue_time_us":186,"thread_start_us":106,"threads_started":1}
Aug 10 18:59:16 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:16 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:16.563795 9623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:16.557782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:34047)
0810 18:59:16.558052 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:16.558065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:16.558168 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:16.558443 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:16.558447 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:16.558638 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:16.558739 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:16.558744 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.559698 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.559703 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:16.560569 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:16.560579 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.561046 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.561050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:16.561133 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:16.561761 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:16.561776 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:16.563603 (+ 1827us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":85,"threads_started":1}
W20260810 18:59:16.563834 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:16.557834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:34047 (local address 127.27.147.190:38623)
0810 18:59:16.557995 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:16.558002 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:16.558143 (+ 141us) server_negotiation.cc:407] Connection header received
0810 18:59:16.558249 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:16.558252 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:16.558299 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:16.558366 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:16.558922 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.559585 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.560686 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.560917 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.561040 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:16.563732 (+ 2692us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:34047: BlockingRecv error: recv got EOF from 127.27.147.130:34047 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Aug 10 18:59:16 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:16 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:16.760289 9624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:16.754365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:50741)
0810 18:59:16.754681 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:16.754694 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:16.754802 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:16.755135 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:16.755139 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:16.755385 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:16.755532 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:16.755540 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.756377 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.756380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:16.756959 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:16.756968 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.757431 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.757435 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:16.757519 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:16.758015 (+ 496us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:16.758026 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:16.760101 (+ 2075us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":121,"threads_started":1}
W20260810 18:59:16.760330 9621 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:16.754488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:50741 (local address 127.27.147.190:38623)
0810 18:59:16.754657 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:16.754662 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:16.754753 (+ 91us) server_negotiation.cc:407] Connection header received
0810 18:59:16.754932 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:16.754934 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:16.754982 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:16.755064 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:16.755652 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.756249 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.757063 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:16.757278 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:16.757354 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:16.760223 (+ 2869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:50741: BlockingRecv error: recv got EOF from 127.27.147.131:50741 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 18:59:17 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:17 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:17.372864 9626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:17.367216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:47111)
0810 18:59:17.367507 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:17.367523 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:17.367613 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:17.367865 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:17.367867 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:17.368048 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:17.368147 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:17.368152 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.368927 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.368929 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:17.369496 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:17.369503 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.369958 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.369960 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:17.370021 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:17.370499 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:17.370511 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:17.372732 (+ 2221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":93,"threads_started":1}
W20260810 18:59:17.373065 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:17.367320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:47111 (local address 127.27.147.190:38623)
0810 18:59:17.367598 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:17.367602 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:17.367616 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:17.367695 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:17.367698 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:17.367737 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:17.367827 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:17.368272 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.368822 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.369610 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.369851 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.369946 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:17.372895 (+ 2949us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:47111: BlockingRecv error: recv got EOF from 127.27.147.129:47111 (error 108)
Metrics: {"server-negotiator.queue_time_us":210,"thread_start_us":124,"threads_started":1}
Aug 10 18:59:17 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:17 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:17.570389 9628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:17.564643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:46627)
0810 18:59:17.564944 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:17.564957 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:17.565060 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:17.565303 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:17.565305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:17.565493 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:17.565593 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:17.565598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.566646 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.566650 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:17.567316 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:17.567327 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.567757 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.567760 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:17.567827 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:17.568409 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:17.568422 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:17.570254 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":105,"threads_started":1}
W20260810 18:59:17.570554 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:17.564727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:46627 (local address 127.27.147.190:38623)
0810 18:59:17.564883 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:17.564887 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:17.565028 (+ 141us) server_negotiation.cc:407] Connection header received
0810 18:59:17.565138 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:17.565140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:17.565186 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:17.565260 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:17.565718 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.566492 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.567429 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.567645 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.567733 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:17.570392 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:46627: BlockingRecv error: recv got EOF from 127.27.147.130:46627 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 18:59:17 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:17 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:17.766542 9630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:17.761158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:50989)
0810 18:59:17.761477 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:17.761490 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:17.761586 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:17.761847 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:17.761850 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:17.762035 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:17.762138 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:17.762144 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.762938 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.762940 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:17.763525 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:17.763534 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.764069 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.764072 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:17.764138 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:17.764633 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:17.764644 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:17.766425 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":132,"threads_started":1}
W20260810 18:59:17.766687 9627 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:17.761188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:50989 (local address 127.27.147.190:38623)
0810 18:59:17.761319 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:17.761324 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:17.761545 (+ 221us) server_negotiation.cc:407] Connection header received
0810 18:59:17.761650 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:17.761652 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:17.761697 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:17.761764 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:17.762250 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.762776 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.763632 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:17.763951 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:17.764035 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:17.766570 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:50989: BlockingRecv error: recv got EOF from 127.27.147.131:50989 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Aug 10 18:59:18 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:18 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:18.379257 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:18.373637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:44753)
0810 18:59:18.373934 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:18.373962 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:18.374068 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:18.374303 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:18.374305 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:18.374490 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:18.374585 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:18.374590 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.375403 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.375405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:18.375975 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:18.375983 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.376398 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.376401 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:18.376461 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:18.376934 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:18.376946 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:18.379132 (+ 2186us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
W20260810 18:59:18.379515 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:18.373784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:44753 (local address 127.27.147.190:38623)
0810 18:59:18.374063 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:18.374067 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:18.374080 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:18.374154 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:18.374156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:18.374196 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:18.374262 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:18.374685 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.375291 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.376071 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.376296 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.376392 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:18.379256 (+ 2864us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:44753: BlockingRecv error: recv got EOF from 127.27.147.129:44753 (error 108)
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":115,"threads_started":1}
Aug 10 18:59:18 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:18 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:18.576996 9634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:18.571238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:50995)
0810 18:59:18.571619 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:18.571633 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:18.571770 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:18.572082 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:18.572085 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:18.572378 (+ 293us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:18.572536 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:18.572545 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.573333 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.573336 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:18.573914 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:18.573922 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.574347 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.574349 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:18.574415 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:18.575003 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:18.575024 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:18.576876 (+ 1852us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":102,"threads_started":1}
W20260810 18:59:18.577143 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:18.571375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:50995 (local address 127.27.147.190:38623)
0810 18:59:18.571512 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:18.571519 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:18.571752 (+ 233us) server_negotiation.cc:407] Connection header received
0810 18:59:18.571881 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:18.571886 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:18.571941 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:18.572015 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:18.572673 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.573229 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.574025 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.574245 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.574325 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:18.577023 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:50995: BlockingRecv error: recv got EOF from 127.27.147.130:50995 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Aug 10 18:59:18 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:18 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:18.772531 9635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:18.767459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:39999)
0810 18:59:18.767764 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:18.767778 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:18.767879 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:18.768068 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:18.768070 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:18.768250 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:18.768341 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:18.768346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.769060 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.769062 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:18.769634 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:18.769641 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.770077 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.770081 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:18.770166 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:18.770625 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:18.770636 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:18.772419 (+ 1783us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":230,"thread_start_us":125,"threads_started":1}
W20260810 18:59:18.772667 9632 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:18.767500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:39999 (local address 127.27.147.190:38623)
0810 18:59:18.767656 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:18.767661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:18.767833 (+ 172us) server_negotiation.cc:407] Connection header received
0810 18:59:18.767949 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:18.767951 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:18.767995 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:18.768057 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:18.768438 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.768969 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.769734 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:18.769958 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:18.770031 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:18.772551 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:39999: BlockingRecv error: recv got EOF from 127.27.147.131:39999 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 18:59:19 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:19 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:19.385977 9636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:19.380044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:58185)
0810 18:59:19.380405 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:19.380421 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:19.380547 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:19.380831 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:19.380835 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:19.381066 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:19.381206 (+ 140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:19.381212 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.381993 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.381996 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:19.382571 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:19.382579 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.383003 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.383005 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:19.383066 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:19.383683 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:19.383695 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:19.385749 (+ 2054us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":278,"thread_start_us":122,"threads_started":1}
W20260810 18:59:19.386104 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:19.380170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:58185 (local address 127.27.147.190:38623)
0810 18:59:19.380425 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:19.380429 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:19.380499 (+ 70us) server_negotiation.cc:407] Connection header received
0810 18:59:19.380635 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:19.380637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:19.380683 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:19.380763 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:19.381331 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.381883 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.382670 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.382906 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.382979 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:19.385945 (+ 2966us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:58185: BlockingRecv error: recv got EOF from 127.27.147.129:58185 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":65,"threads_started":1}
Aug 10 18:59:19 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:19 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:19.583710 9638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:19.577914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:58587)
0810 18:59:19.578186 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:19.578200 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:19.578297 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:19.578525 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:19.578527 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:19.578713 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:19.578839 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:19.578845 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.579626 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.579628 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:19.580218 (+ 590us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:19.580226 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.580893 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.580897 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:19.580995 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:19.581631 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:19.581645 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:19.583539 (+ 1894us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":101,"threads_started":1}
W20260810 18:59:19.583774 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:19.578034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:58587 (local address 127.27.147.190:38623)
0810 18:59:19.578181 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:19.578185 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:19.578291 (+ 106us) server_negotiation.cc:407] Connection header received
0810 18:59:19.578378 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:19.578381 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:19.578427 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:19.578488 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:19.578967 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.579527 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.580339 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.580787 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.580888 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:19.583668 (+ 2780us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:58587: BlockingRecv error: recv got EOF from 127.27.147.130:58587 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Aug 10 18:59:19 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:19 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:19.779137 9639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:19.773459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:43307)
0810 18:59:19.773751 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:19.773765 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:19.773866 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:19.774093 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:19.774095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:19.774299 (+ 204us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:19.774399 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:19.774404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.775488 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.775491 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:19.776118 (+ 627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:19.776128 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.776591 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.776593 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:19.776655 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:19.777179 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:19.777191 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:19.779021 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":106,"threads_started":1}
W20260810 18:59:19.779358 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:19.773536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:43307 (local address 127.27.147.190:38623)
0810 18:59:19.773713 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:19.773718 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:19.773822 (+ 104us) server_negotiation.cc:407] Connection header received
0810 18:59:19.773945 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:19.773947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:19.773993 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:19.774053 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:19.774501 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.775345 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.776237 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:19.776487 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:19.776572 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:19.779216 (+ 2644us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:43307: BlockingRecv error: recv got EOF from 127.27.147.131:43307 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
Aug 10 18:59:20 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:20 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:20.392370 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:20.386886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:54211)
0810 18:59:20.387191 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:20.387204 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:20.387328 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:20.387551 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:20.387553 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:20.387740 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:20.387839 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:20.387844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.388571 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.388573 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:20.389141 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:20.389149 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.389606 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.389608 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:20.389672 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:20.390231 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:20.390244 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:20.392250 (+ 2006us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
W20260810 18:59:20.392710 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:20.386920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:54211 (local address 127.27.147.190:38623)
0810 18:59:20.387175 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:20.387178 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:20.387255 (+ 77us) server_negotiation.cc:407] Connection header received
0810 18:59:20.387394 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:20.387396 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:20.387438 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:20.387517 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:20.387940 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.388478 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.389270 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.389517 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.389587 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:20.392563 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:54211: BlockingRecv error: recv got EOF from 127.27.147.129:54211 (error 108)
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":96,"threads_started":1}
Aug 10 18:59:20 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:20 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:20.594699 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:20.584588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:35753)
0810 18:59:20.584923 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:20.584936 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:20.585035 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:20.585284 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:20.585286 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:20.585476 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:20.585574 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:20.585580 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.586579 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.586583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:20.587441 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:20.587453 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.587897 (+ 444us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.587900 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:20.587967 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:20.592730 (+ 4763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:20.592744 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:20.594570 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":94,"threads_started":1}
W20260810 18:59:20.594852 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:20.584653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:35753 (local address 127.27.147.190:38623)
0810 18:59:20.584838 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:20.584843 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:20.585004 (+ 161us) server_negotiation.cc:407] Connection header received
0810 18:59:20.585123 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:20.585125 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:20.585172 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:20.585241 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:20.585679 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.586434 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.587549 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.587761 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.587850 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:20.594705 (+ 6855us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:35753: BlockingRecv error: recv got EOF from 127.27.147.130:35753 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 18:59:20 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:20 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:20.785966 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:20.779896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:57645)
0810 18:59:20.780207 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:20.780220 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:20.780346 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:20.780699 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:20.780701 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:20.780972 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:20.781107 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:20.781115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.781975 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.781977 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:20.782539 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:20.782547 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.783043 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.783046 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:20.783111 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:20.783687 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:20.783700 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:20.785788 (+ 2088us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":82,"threads_started":1}
W20260810 18:59:20.786041 9642 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:20.779970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:57645 (local address 127.27.147.190:38623)
0810 18:59:20.780120 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:20.780124 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:20.780298 (+ 174us) server_negotiation.cc:407] Connection header received
0810 18:59:20.780484 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:20.780488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:20.780570 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:20.780654 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:20.781231 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.781880 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.782666 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:20.782920 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:20.783009 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:20.785927 (+ 2918us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:57645: BlockingRecv error: recv got EOF from 127.27.147.131:57645 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Aug 10 18:59:21 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:21 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:21.398989 9646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:21.393197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:50485)
0810 18:59:21.393579 (+ 382us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:21.393598 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:21.393693 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:21.394031 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:21.394034 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:21.394229 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:21.394361 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:21.394368 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.395236 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.395239 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:21.395826 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:21.395836 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.396314 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.396317 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:21.396380 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:21.396842 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:21.396853 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:21.398807 (+ 1954us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":287,"thread_start_us":83,"threads_started":1}
W20260810 18:59:21.399133 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:21.393311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:50485 (local address 127.27.147.190:38623)
0810 18:59:21.393586 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:21.393592 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:21.393655 (+ 63us) server_negotiation.cc:407] Connection header received
0810 18:59:21.393797 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:21.393800 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:21.393878 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:21.393981 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:21.394477 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.395110 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.395931 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.396213 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.396291 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:21.399000 (+ 2709us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:50485: BlockingRecv error: recv got EOF from 127.27.147.129:50485 (error 108)
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":65,"threads_started":1}
Aug 10 18:59:21 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:21 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:21.601449 9651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:21.595662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:55215)
0810 18:59:21.595960 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:21.595974 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:21.596089 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:21.596342 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:21.596344 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:21.596531 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:21.596630 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:21.596635 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.597509 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.597513 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:21.598083 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:21.598092 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.598541 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.598544 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:21.598617 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:21.599133 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:21.599154 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:21.601331 (+ 2177us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":226,"thread_start_us":117,"threads_started":1}
W20260810 18:59:21.601583 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:21.595715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:55215 (local address 127.27.147.190:38623)
0810 18:59:21.595872 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:21.595877 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:21.596049 (+ 172us) server_negotiation.cc:407] Connection header received
0810 18:59:21.596161 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:21.596163 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:21.596212 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:21.596277 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:21.596735 (+ 458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.597401 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.598195 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.598411 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.598488 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:21.601464 (+ 2976us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:55215: BlockingRecv error: recv got EOF from 127.27.147.130:55215 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 18:59:21 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:21 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:21.792276 9652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:21.786741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:33063)
0810 18:59:21.787063 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:21.787077 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:21.787174 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:21.787530 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:21.787532 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:21.787712 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:21.787815 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:21.787820 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.788562 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.788564 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:21.789123 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:21.789131 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.789590 (+ 459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.789593 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:21.789682 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:21.790281 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:21.790293 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:21.792159 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":236,"thread_start_us":147,"threads_started":1}
W20260810 18:59:21.792472 9647 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:21.786796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:33063 (local address 127.27.147.190:38623)
0810 18:59:21.786949 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:21.786953 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:21.787145 (+ 192us) server_negotiation.cc:407] Connection header received
0810 18:59:21.787357 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:21.787360 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:21.787413 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:21.787493 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:21.787918 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.788464 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.789238 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:21.789486 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:21.789568 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:21.792330 (+ 2762us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:33063: BlockingRecv error: recv got EOF from 127.27.147.131:33063 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Aug 10 18:59:22 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:22 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:22.406800 9653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:22.399828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:35791)
0810 18:59:22.400260 (+ 432us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:22.400280 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:22.400422 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:22.400712 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:22.400717 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:22.400975 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:22.401123 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:22.401132 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.402344 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.402350 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:22.403261 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:22.403279 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.403907 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.403911 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:22.404019 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:22.404633 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:22.404653 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:22.406683 (+ 2030us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":150,"threads_started":1}
W20260810 18:59:22.407037 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:22.399913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:35791 (local address 127.27.147.190:38623)
0810 18:59:22.400276 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:22.400282 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:22.400436 (+ 154us) server_negotiation.cc:407] Connection header received
0810 18:59:22.400520 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:22.400522 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:22.400582 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:22.400677 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:22.401238 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.402205 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.403387 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.403757 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.403878 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:22.406870 (+ 2992us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:35791: BlockingRecv error: recv got EOF from 127.27.147.129:35791 (error 108)
Metrics: {"server-negotiator.queue_time_us":272,"thread_start_us":84,"threads_started":1}
Aug 10 18:59:22 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:22 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:22.608333 9657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:22.602351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:36227)
0810 18:59:22.602660 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:22.602674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:22.602815 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:22.603117 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:22.603120 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:22.603350 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:22.603446 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:22.603451 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.604214 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.604219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:22.604790 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:22.604799 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.605298 (+ 499us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.605302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:22.605392 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:22.606036 (+ 644us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:22.606053 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:22.608158 (+ 2105us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":232,"thread_start_us":96,"threads_started":1}
W20260810 18:59:22.608397 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:22.602422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:36227 (local address 127.27.147.190:38623)
0810 18:59:22.602588 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:22.602593 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:22.602762 (+ 169us) server_negotiation.cc:407] Connection header received
0810 18:59:22.602943 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:22.602946 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:22.603003 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:22.603092 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:22.603566 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.604100 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.604926 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.605155 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.605283 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:22.608290 (+ 3007us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:36227: BlockingRecv error: recv got EOF from 127.27.147.130:36227 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Aug 10 18:59:22 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:22 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:22.798717 9658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:22.793028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:51479)
0810 18:59:22.793348 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:22.793369 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:22.793523 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:22.793855 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:22.793858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:22.794077 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:22.794171 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:22.794176 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.795032 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.795039 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:22.795689 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:22.795699 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.796133 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.796136 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:22.796203 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:22.796756 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:22.796768 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:22.798603 (+ 1835us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":90,"threads_started":1}
W20260810 18:59:22.798879 9654 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:22.793141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:51479 (local address 127.27.147.190:38623)
0810 18:59:22.793305 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:22.793310 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:22.793448 (+ 138us) server_negotiation.cc:407] Connection header received
0810 18:59:22.793647 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:22.793652 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:22.793704 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:22.793790 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:22.794294 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.794898 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.795799 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:22.796026 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:22.796108 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:22.798740 (+ 2632us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:51479: BlockingRecv error: recv got EOF from 127.27.147.131:51479 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 18:59:23 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:23 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:23.413599 9659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:23.407666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:34871)
0810 18:59:23.408088 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:23.408104 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:23.408241 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:23.408583 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:23.408586 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:23.408836 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:23.408970 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:23.408979 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.409746 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.409748 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:23.410327 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:23.410335 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.410781 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.410783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:23.410919 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:23.411462 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:23.411481 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:23.413456 (+ 1975us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":81,"threads_started":1}
W20260810 18:59:23.413784 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:23.407801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:34871 (local address 127.27.147.190:38623)
0810 18:59:23.408063 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:23.408073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:23.408191 (+ 118us) server_negotiation.cc:407] Connection header received
0810 18:59:23.408345 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:23.408348 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:23.408415 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:23.408522 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:23.409098 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.409647 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.410453 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.410679 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.410764 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:23.413602 (+ 2838us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:34871: BlockingRecv error: recv got EOF from 127.27.147.129:34871 (error 108)
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":60,"threads_started":1}
Aug 10 18:59:23 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:23 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:23.614346 9661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:23.609180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:53229)
0810 18:59:23.609472 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:23.609486 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:23.609583 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:23.609807 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:23.609809 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:23.609991 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:23.610089 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:23.610094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.610808 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.610846 (+ 38us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:23.611396 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:23.611404 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.611815 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.611818 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:23.611877 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:23.612350 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:23.612362 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:23.614151 (+ 1789us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":106,"threads_started":1}
W20260810 18:59:23.614405 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:23.609242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:53229 (local address 127.27.147.190:38623)
0810 18:59:23.609392 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:23.609404 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:23.609558 (+ 154us) server_negotiation.cc:407] Connection header received
0810 18:59:23.609652 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:23.609655 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:23.609700 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:23.609766 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:23.610185 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.610714 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.611500 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.611718 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.611796 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:23.614292 (+ 2496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:53229: BlockingRecv error: recv got EOF from 127.27.147.130:53229 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 18:59:23 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:23 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:23.804826 9662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:23.799604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:56185)
0810 18:59:23.799901 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:23.799914 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:23.800006 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:23.800242 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:23.800244 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:23.800431 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:23.800530 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:23.800535 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.801255 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.801257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:23.801819 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:23.801827 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.802241 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.802244 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:23.802311 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:23.802911 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:23.802924 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:23.804704 (+ 1780us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":115,"threads_started":1}
W20260810 18:59:23.805002 9660 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:23.799782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:56185 (local address 127.27.147.190:38623)
0810 18:59:23.799919 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:23.799923 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:23.799986 (+ 63us) server_negotiation.cc:407] Connection header received
0810 18:59:23.800086 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:23.800089 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:23.800134 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:23.800201 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:23.800634 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.801161 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.801919 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:23.802142 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:23.802219 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:23.804836 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:56185: BlockingRecv error: recv got EOF from 127.27.147.131:56185 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Aug 10 18:59:24 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:24 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:24.420148 9663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:24.414487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:34539)
0810 18:59:24.414775 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:24.414792 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:24.414928 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:24.415292 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:24.415295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:24.415559 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:24.415726 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:24.415739 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.416530 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.416532 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:24.417079 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:24.417087 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.417513 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.417515 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:24.417578 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:24.418055 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:24.418066 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:24.420031 (+ 1965us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":101,"threads_started":1}
W20260810 18:59:24.420433 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:24.414547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:34539 (local address 127.27.147.190:38623)
0810 18:59:24.414868 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:24.414872 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:24.414890 (+ 18us) server_negotiation.cc:407] Connection header received
0810 18:59:24.415052 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:24.415055 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:24.415135 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:24.415240 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:24.415869 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.416428 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.417178 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.417408 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.417483 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:24.420172 (+ 2689us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:34539: BlockingRecv error: recv got EOF from 127.27.147.129:34539 (error 108)
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
Aug 10 18:59:24 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:24 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:24.620720 9669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:24.615158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:60553)
0810 18:59:24.615462 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:24.615490 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:24.615642 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:24.615922 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:24.615925 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:24.616147 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:24.616243 (+ 96us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:24.616249 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.617028 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.617033 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:24.617602 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:24.617610 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.618099 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.618103 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:24.618191 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:24.618694 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:24.618706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:24.620548 (+ 1842us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":91,"threads_started":1}
W20260810 18:59:24.620790 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:24.615236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:60553 (local address 127.27.147.190:38623)
0810 18:59:24.615452 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:24.615457 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:24.615579 (+ 122us) server_negotiation.cc:407] Connection header received
0810 18:59:24.615745 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:24.615749 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:24.615798 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:24.615898 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:24.616366 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.616909 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.617727 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.617979 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.618093 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:24.620680 (+ 2587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:60553: BlockingRecv error: recv got EOF from 127.27.147.130:60553 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
Aug 10 18:59:24 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:24 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:24.811647 9670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:24.805669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:43305)
0810 18:59:24.805990 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:24.806008 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:24.806132 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:24.806380 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:24.806383 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:24.806571 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:24.806668 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:24.806673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.807498 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.807501 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:24.808105 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:24.808114 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.808866 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.808869 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:24.808938 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:24.809474 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:24.809486 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:24.811452 (+ 1966us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":114,"threads_started":1}
W20260810 18:59:24.811684 9667 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:24.805740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:43305 (local address 127.27.147.190:38623)
0810 18:59:24.805894 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:24.805899 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:24.806111 (+ 212us) server_negotiation.cc:407] Connection header received
0810 18:59:24.806217 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:24.806220 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:24.806270 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:24.806337 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:24.806778 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.807384 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.808228 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:24.808718 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:24.808817 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:24.811575 (+ 2758us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:43305: BlockingRecv error: recv got EOF from 127.27.147.131:43305 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 18:59:25 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:25 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:25.427033 9673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:25.420908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:50533)
0810 18:59:25.421218 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:25.421231 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:25.421333 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:25.421589 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:25.421591 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:25.421808 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:25.421926 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:25.421932 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.423022 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.423026 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:25.423937 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:25.423948 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.424386 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.424389 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:25.424466 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:25.424961 (+ 495us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:25.424973 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:25.426914 (+ 1941us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":86,"threads_started":1}
W20260810 18:59:25.427210 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:25.421050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:50533 (local address 127.27.147.190:38623)
0810 18:59:25.421271 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:25.421274 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:25.421287 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:25.421426 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:25.421428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:25.421471 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:25.421548 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:25.422129 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.422897 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.424050 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.424273 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.424355 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:25.427053 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:50533: BlockingRecv error: recv got EOF from 127.27.147.129:50533 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":58,"threads_started":1}
Aug 10 18:59:25 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:25 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:25.632332 9675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:25.621751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:46961)
0810 18:59:25.622034 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:25.622047 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:25.622147 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:25.622382 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:25.622384 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:25.622586 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:25.622689 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:25.622694 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.623481 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.623484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:25.624038 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:25.624046 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.624481 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.624484 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:25.624549 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:25.630211 (+ 5662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:25.630225 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:25.632206 (+ 1981us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":96,"threads_started":1}
W20260810 18:59:25.632469 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:25.621847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:46961 (local address 127.27.147.190:38623)
0810 18:59:25.621985 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:25.621989 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:25.622117 (+ 128us) server_negotiation.cc:407] Connection header received
0810 18:59:25.622225 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:25.622227 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:25.622272 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:25.622343 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:25.622790 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.623365 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.624142 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.624374 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.624451 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:25.632350 (+ 7899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:46961: BlockingRecv error: recv got EOF from 127.27.147.130:46961 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Aug 10 18:59:25 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:25 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:25.817842 9676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:25.812413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:44459)
0810 18:59:25.812715 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:25.812729 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:25.812837 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:25.813084 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:25.813086 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:25.813280 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:25.813380 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:25.813385 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.814130 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.814133 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:25.814715 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:25.814724 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.815238 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.815241 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:25.815310 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:25.815832 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:25.815844 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:25.817703 (+ 1859us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":221,"thread_start_us":110,"threads_started":1}
W20260810 18:59:25.817974 9674 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:25.812510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:44459 (local address 127.27.147.190:38623)
0810 18:59:25.812657 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:25.812661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:25.812807 (+ 146us) server_negotiation.cc:407] Connection header received
0810 18:59:25.812917 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:25.812919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:25.812967 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:25.813038 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:25.813484 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.814018 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.814862 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:25.815086 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:25.815186 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:25.817852 (+ 2666us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:44459: BlockingRecv error: recv got EOF from 127.27.147.131:44459 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Aug 10 18:59:26 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:26 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:26.433458 9679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:26.427811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:33693)
0810 18:59:26.428162 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:26.428176 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:26.428265 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:26.428507 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:26.428509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:26.428705 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:26.428828 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:26.428835 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.429589 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.429591 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:26.430162 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:26.430170 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.430596 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.430599 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:26.430670 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:26.431186 (+ 516us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:26.431198 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:26.433344 (+ 2146us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":188,"threads_started":1}
W20260810 18:59:26.433648 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:26.427906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:33693 (local address 127.27.147.190:38623)
0810 18:59:26.428130 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:26.428133 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:26.428244 (+ 111us) server_negotiation.cc:407] Connection header received
0810 18:59:26.428332 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:26.428334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:26.428393 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:26.428474 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:26.428939 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.429492 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.430261 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.430487 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.430560 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:26.433479 (+ 2919us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:33693: BlockingRecv error: recv got EOF from 127.27.147.129:33693 (error 108)
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":62,"threads_started":1}
Aug 10 18:59:26 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:26 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:26.639297 9681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:26.633388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:37467)
0810 18:59:26.633726 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:26.633743 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:26.633901 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:26.634191 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:26.634194 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:26.634406 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:26.634512 (+ 106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:26.634519 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.635362 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.635367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:26.635934 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:26.635943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.636414 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.636418 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:26.636523 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:26.637038 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:26.637050 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:26.639126 (+ 2076us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":104,"threads_started":1}
W20260810 18:59:26.639448 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:26.633560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:37467 (local address 127.27.147.190:38623)
0810 18:59:26.633729 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:26.633735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:26.633835 (+ 100us) server_negotiation.cc:407] Connection header received
0810 18:59:26.634013 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:26.634016 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:26.634071 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:26.634161 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:26.634638 (+ 477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.635249 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.636063 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.636295 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.636406 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:26.639310 (+ 2904us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:37467: BlockingRecv error: recv got EOF from 127.27.147.130:37467 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 10 18:59:26 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:26 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:26.824443 9682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:26.818885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:38821)
0810 18:59:26.819162 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:26.819175 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:26.819299 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:26.819531 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:26.819534 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:26.819722 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:26.819821 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:26.819826 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.820537 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.820540 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:26.821089 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:26.821096 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.821493 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.821496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:26.821554 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:26.822136 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:26.822152 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:26.824328 (+ 2176us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":103,"threads_started":1}
W20260810 18:59:26.824615 9680 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:26.818955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:38821 (local address 127.27.147.190:38623)
0810 18:59:26.819109 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:26.819112 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:26.819228 (+ 116us) server_negotiation.cc:407] Connection header received
0810 18:59:26.819368 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:26.819371 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:26.819414 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:26.819487 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:26.819920 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.820433 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.821188 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:26.821399 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:26.821480 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:26.824453 (+ 2973us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:38821: BlockingRecv error: recv got EOF from 127.27.147.131:38821 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 10 18:59:27 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:27 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:27.439419 9685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:27.434321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:45841)
0810 18:59:27.434605 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:27.434618 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:27.434697 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:27.434942 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:27.434944 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:27.435117 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:27.435210 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:27.435215 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.435953 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.435956 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:27.436516 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:27.436524 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.436969 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.436972 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:27.437036 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:27.437488 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:27.437499 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:27.439303 (+ 1804us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":102,"threads_started":1}
W20260810 18:59:27.439576 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:27.434373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:45841 (local address 127.27.147.190:38623)
0810 18:59:27.434674 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:27.434677 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:27.434691 (+ 14us) server_negotiation.cc:407] Connection header received
0810 18:59:27.434764 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:27.434766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:27.434847 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:27.434932 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:27.435321 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.435848 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.436626 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.436852 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.436932 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:27.439445 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:45841: BlockingRecv error: recv got EOF from 127.27.147.129:45841 (error 108)
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":89,"threads_started":1}
Aug 10 18:59:27 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:27 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:27.645812 9689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:27.640080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:52091)
0810 18:59:27.640379 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:27.640394 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:27.640543 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:27.640825 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:27.640829 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:27.641055 (+ 226us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:27.641150 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:27.641156 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.641942 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.641947 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:27.642544 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:27.642554 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.643091 (+ 537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.643095 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:27.643186 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:27.643694 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:27.643706 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:27.645615 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":95,"threads_started":1}
W20260810 18:59:27.645850 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:27.640147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:52091 (local address 127.27.147.190:38623)
0810 18:59:27.640328 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:27.640333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:27.640484 (+ 151us) server_negotiation.cc:407] Connection header received
0810 18:59:27.640653 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:27.640657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:27.640710 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:27.640801 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:27.641281 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.641827 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.642672 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.642971 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.643084 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:27.645743 (+ 2659us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:52091: BlockingRecv error: recv got EOF from 127.27.147.130:52091 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Aug 10 18:59:27 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:27 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.131@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:27.831172 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:27.825299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.131:50993)
0810 18:59:27.825612 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:27.825625 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:27.825735 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:27.826002 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:27.826004 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:27.826194 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:27.826368 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:27.826375 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.827173 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.827176 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:27.827759 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:27.827768 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.828172 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.828175 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:27.828238 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:27.828875 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:27.828890 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:27.831056 (+ 2166us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":229,"thread_start_us":106,"threads_started":1}
W20260810 18:59:27.831336 9688 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:27.825419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:50993 (local address 127.27.147.190:38623)
0810 18:59:27.825606 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:27.825613 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:27.825697 (+ 84us) server_negotiation.cc:407] Connection header received
0810 18:59:27.825835 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:27.825838 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:27.825886 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:27.825954 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:27.826482 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.827074 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.827859 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:27.828075 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:27.828150 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:27.831182 (+ 3032us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.131:50993: BlockingRecv error: recv got EOF from 127.27.147.131:50993 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
Aug 10 18:59:28 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:28 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.129@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:28.445792 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:28.440315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.129:33151)
0810 18:59:28.440609 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:28.440623 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:28.440702 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:28.440925 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:28.440927 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:28.441103 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:28.441200 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:28.441205 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.441921 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.441923 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:28.442483 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:28.442490 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.443056 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.443059 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:28.443126 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:28.443641 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:28.443653 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:28.445610 (+ 1957us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":87,"threads_started":1}
W20260810 18:59:28.445889 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:28.440420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:33151 (local address 127.27.147.190:38623)
0810 18:59:28.440662 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:28.440665 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:28.440678 (+ 13us) server_negotiation.cc:407] Connection header received
0810 18:59:28.440781 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:28.440783 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:28.440825 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:28.440898 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:28.441304 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.441832 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.442582 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.442800 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.442906 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:28.445737 (+ 2831us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.129:33151: BlockingRecv error: recv got EOF from 127.27.147.129:33151 (error 108)
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":59,"threads_started":1}
Aug 10 18:59:28 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
Aug 10 18:59:28 dist-test-slave-znh6 krb5kdc[8057](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.27.147.130@KRBTEST.COM for oryx/127.27.147.190@KRBTEST.COM, Server not found in Kerberos database
I20260810 18:59:28.652271 9696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:28.646712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38623 (local address 127.27.147.130:44945)
0810 18:59:28.647070 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:28.647087 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:28.647209 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:28.647496 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:28.647499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:28.647718 (+ 219us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:28.647811 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:28.647817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.648585 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.648588 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:28.649188 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:28.649197 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.649671 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.649675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:28.649761 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:28.650263 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:28.650274 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:28.652137 (+ 1863us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.27.147.190:38623: Server oryx/127.27.147.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":257,"thread_start_us":111,"threads_started":1}
W20260810 18:59:28.652421 9692 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:28.646842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:44945 (local address 127.27.147.190:38623)
0810 18:59:28.646994 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:28.646999 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:28.647178 (+ 179us) server_negotiation.cc:407] Connection header received
0810 18:59:28.647334 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:28.647337 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:28.647382 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:28.647476 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:28.647927 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.648462 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.649313 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.649552 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.649666 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:28.652271 (+ 2605us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.27.147.130:44945: BlockingRecv error: recv got EOF from 127.27.147.130:44945 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:59:28.726250 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 8962
I20260810 18:59:28.749899 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 9189
I20260810 18:59:28.761075 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 9404
I20260810 18:59:28.772917 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 8850
2026-08-10T18:59:28Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (18305 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T18:59:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:59:28Z Disabled control of system clock
I20260810 18:59:28.810178 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35677
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:35677
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 18:59:28.921902 9701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:28.922132 9701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:28.922165 9701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:28.926157 9701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:28.926208 9701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:28.926223 9701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:28.926240 9701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:28.926258 9701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:28.930835 9701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:35677
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:28.931820 9701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:28.933153 9701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:28.941880 9714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:28.942008 9701 server_base.cc:1034] running on GCE node
W20260810 18:59:28.941877 9708 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:28.941891 9710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:28.942710 9701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:28.943823 9701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:28.945016 9701 hybrid_clock.cc:648] HybridClock initialized: now 1786388368944988 us; error 43 us; skew 500 ppm
I20260810 18:59:28.946777 9701 webserver.cc:460] Webserver started at http://127.27.147.190:40601/ using document root <none> and password file <none>
I20260810 18:59:28.947376 9701 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:28.947481 9701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:28.947717 9701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:28.949707 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "b655112e1f3447f6baec1cca469e9f86"
format_stamp: "Formatted at 2026-08-10 18:59:28 on dist-test-slave-znh6"
server_key: "20ddf9956c844d004dfffc27f41a45f0"
server_key_iv: "65888ce70c1ed80876ef9247ad2f910f"
server_key_version: "encryptionkey@0"
I20260810 18:59:28.950274 9701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "b655112e1f3447f6baec1cca469e9f86"
format_stamp: "Formatted at 2026-08-10 18:59:28 on dist-test-slave-znh6"
server_key: "20ddf9956c844d004dfffc27f41a45f0"
server_key_iv: "65888ce70c1ed80876ef9247ad2f910f"
server_key_version: "encryptionkey@0"
I20260810 18:59:28.953850 9701 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 18:59:28.956300 9726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:28.957494 9701 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:59:28.957634 9701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "b655112e1f3447f6baec1cca469e9f86"
format_stamp: "Formatted at 2026-08-10 18:59:28 on dist-test-slave-znh6"
server_key: "20ddf9956c844d004dfffc27f41a45f0"
server_key_iv: "65888ce70c1ed80876ef9247ad2f910f"
server_key_version: "encryptionkey@0"
I20260810 18:59:28.957746 9701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:28.971095 9701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:28.971688 9701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:28.971875 9701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:28.978367 9701 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:35677
I20260810 18:59:28.978374 9800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:35677 every 8 connection(s)
I20260810 18:59:28.979300 9701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:59:28.982331 9801 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:28.985872 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 9701
I20260810 18:59:28.986011 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:59:28.986357 28238 external_mini_cluster.cc:1428] Setting key 0af7d3bf46ae672a67d5d60dde306fda
I20260810 18:59:28.988292 9801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86: Bootstrap starting.
I20260810 18:59:28.990438 9801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:28.991212 9801 log.cc:824] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:28.992913 9801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86: No bootstrap required, opened a new log
I20260810 18:59:28.995575 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:28.987748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50190 (local address 127.27.147.190:35677)
0810 18:59:28.988455 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:28.988479 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:28.988515 (+ 36us) server_negotiation.cc:407] Connection header received
0810 18:59:28.989141 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:28.989157 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:28.989520 (+ 363us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:28.989839 (+ 319us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:28.990278 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.991408 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.992299 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:28.992750 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:28.992922 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:28.993749 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:28.993754 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:28.993766 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:28.994129 (+ 363us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:28.994181 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:28.994200 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:28.994314 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:28.994439 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 18:59:28.994683 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":137,"threads_started":1}
I20260810 18:59:28.996173 9801 raft_consensus.cc:348] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b655112e1f3447f6baec1cca469e9f86" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35677 } }
I20260810 18:59:28.996402 9801 raft_consensus.cc:374] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:28.996467 9801 raft_consensus.cc:729] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b655112e1f3447f6baec1cca469e9f86, State: Initialized, Role: FOLLOWER
I20260810 18:59:28.996870 9801 consensus_queue.cc:260] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b655112e1f3447f6baec1cca469e9f86" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35677 } }
I20260810 18:59:28.997026 9801 raft_consensus.cc:388] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:59:28.997110 9801 raft_consensus.cc:482] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:59:28.997236 9801 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:28.998152 9801 raft_consensus.cc:504] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b655112e1f3447f6baec1cca469e9f86" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35677 } }
I20260810 18:59:28.998479 9801 leader_election.cc:302] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b655112e1f3447f6baec1cca469e9f86; no voters:
I20260810 18:59:28.998773 9801 leader_election.cc:288] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:59:28.998926 9809 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:28.999166 9809 raft_consensus.cc:686] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [term 1 LEADER]: Becoming Leader. State: Replica: b655112e1f3447f6baec1cca469e9f86, State: Running, Role: LEADER
I20260810 18:59:28.999464 9809 consensus_queue.cc:237] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b655112e1f3447f6baec1cca469e9f86" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35677 } }
I20260810 18:59:28.999977 9801 sys_catalog.cc:564] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:29.001540 9813 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b655112e1f3447f6baec1cca469e9f86. Latest consensus state: current_term: 1 leader_uuid: "b655112e1f3447f6baec1cca469e9f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b655112e1f3447f6baec1cca469e9f86" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35677 } } }
I20260810 18:59:29.001672 9813 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:29.001528 9812 sys_catalog.cc:455] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b655112e1f3447f6baec1cca469e9f86" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b655112e1f3447f6baec1cca469e9f86" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35677 } } }
I20260810 18:59:29.001735 9812 sys_catalog.cc:458] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:29.002077 9833 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:59:29.005237 9833 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:59:29.010459 9833 catalog_manager.cc:1347] Generated new cluster ID: f4d404d90ade4497851aa5a71038fff6
I20260810 18:59:29.010531 9833 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:59:29.041163 9833 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:59:29.042371 9833 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:59:29.048223 9833 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b655112e1f3447f6baec1cca469e9f86: Generated new TSK 0
I20260810 18:59:29.048986 9833 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:59:29.054113 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35677
--builtin_ntp_servers=127.27.147.148:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 18:59:29.167052 9845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:29.167275 9845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:29.167310 9845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:29.171912 9845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:29.171959 9845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:29.172036 9845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:59:29.176594 9845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9845
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:29.177551 9845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:29.178910 9845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:29.188295 9857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:29.188356 9845 server_base.cc:1034] running on GCE node
W20260810 18:59:29.188277 9859 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:29.188277 9862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:29.188896 9845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:29.189560 9845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:29.190774 9845 hybrid_clock.cc:648] HybridClock initialized: now 1786388369190747 us; error 29 us; skew 500 ppm
I20260810 18:59:29.192516 9845 webserver.cc:460] Webserver started at http://127.27.147.129:45051/ using document root <none> and password file <none>
I20260810 18:59:29.193089 9845 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:29.193181 9845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:29.193426 9845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:29.195463 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "8ea4b9ba34bd41dda3ebae3451421467"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "dece76f362998928ffb5c25f183bf78d"
server_key_iv: "9617678d7cc58761bbb3e2705868e1f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.196018 9845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "8ea4b9ba34bd41dda3ebae3451421467"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "dece76f362998928ffb5c25f183bf78d"
server_key_iv: "9617678d7cc58761bbb3e2705868e1f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.199609 9845 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:59:29.202122 9870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.203476 9845 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:29.203620 9845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "8ea4b9ba34bd41dda3ebae3451421467"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "dece76f362998928ffb5c25f183bf78d"
server_key_iv: "9617678d7cc58761bbb3e2705868e1f1"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.203734 9845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:29.230670 9845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:29.231434 9845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:29.231647 9845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:29.232251 9845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:29.233381 9845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:29.233454 9845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.233525 9845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:29.233578 9845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.242620 9845 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:40321
I20260810 18:59:29.242651 10047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:40321 every 8 connection(s)
I20260810 18:59:29.243618 9845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:59:29.251847 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 9845
I20260810 18:59:29.251976 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:59:29.252351 28238 external_mini_cluster.cc:1428] Setting key f4e45cd948b3a302d59fe8753211dda7
I20260810 18:59:29.252890 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:29.245512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:55833 (local address 127.27.147.190:35677)
0810 18:59:29.245653 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:29.245657 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:29.246712 (+ 1055us) server_negotiation.cc:407] Connection header received
0810 18:59:29.248166 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:29.248170 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:29.248226 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:29.248307 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:29.249154 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.249673 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.250808 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.251111 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.251207 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:29.252393 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:29.252398 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:29.252401 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:29.252418 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:29.252444 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:29.252456 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:29.252537 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:29.252687 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 18:59:29.252753 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:59:29.253780 10054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:29.245908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35677 (local address 127.27.147.129:55833)
0810 18:59:29.246567 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:29.246608 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:29.247942 (+ 1334us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:29.248465 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:29.248472 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:29.248550 (+ 78us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:29.249031 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:29.249043 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.249791 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.249793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:29.250664 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:29.250673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.251256 (+ 583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.251259 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:29.251394 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:29.252128 (+ 734us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:59:29.252129 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:29.252170 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:59:29.252236 (+ 66us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:29.252538 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:29.252546 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:29.252550 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:59:29.252652 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 18:59:29.252930 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":115,"threads_started":1}
I20260810 18:59:29.254443 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35677
--builtin_ntp_servers=127.27.147.148:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:59:29.255160 10048 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35677
I20260810 18:59:29.255409 10048 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:29.256016 10048 heartbeater.cc:511] Master 127.27.147.190:35677 requested a full tablet report, sending...
I20260810 18:59:29.257615 9749 ts_manager.cc:194] Registered new tserver with Master: 8ea4b9ba34bd41dda3ebae3451421467 (127.27.147.129:40321)
I20260810 18:59:29.259325 9749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.129:55833
W20260810 18:59:29.369129 10055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:29.369369 10055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:29.369403 10055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:29.373271 10055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:29.373317 10055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:29.373365 10055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:59:29.377835 10055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:29.378762 10055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:29.380182 10055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:29.389792 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:29.389838 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:29.389884 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:29.390074 10055 server_base.cc:1034] running on GCE node
I20260810 18:59:29.390568 10055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:29.391283 10055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:29.392477 10055 hybrid_clock.cc:648] HybridClock initialized: now 1786388369392464 us; error 51 us; skew 500 ppm
I20260810 18:59:29.394374 10055 webserver.cc:460] Webserver started at http://127.27.147.130:36061/ using document root <none> and password file <none>
I20260810 18:59:29.395032 10055 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:29.395126 10055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:29.395365 10055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:29.397390 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "ff1857b5c8bd472e9ce85b5f2dc4291b"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "ee84dab16e32bea9090dd132ce5f2428"
server_key_iv: "45b76de9a3653c35777df2d79164602a"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.397955 10055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "ff1857b5c8bd472e9ce85b5f2dc4291b"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "ee84dab16e32bea9090dd132ce5f2428"
server_key_iv: "45b76de9a3653c35777df2d79164602a"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.401994 10055 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 18:59:29.404678 10074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.405893 10055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 18:59:29.406028 10055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "ff1857b5c8bd472e9ce85b5f2dc4291b"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "ee84dab16e32bea9090dd132ce5f2428"
server_key_iv: "45b76de9a3653c35777df2d79164602a"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.406134 10055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:29.449012 10055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:29.449771 10055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:29.449975 10055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:29.450603 10055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:29.451704 10055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:29.451753 10055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.451793 10055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:29.451849 10055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.460995 10055 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:38041
I20260810 18:59:29.461028 10247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:38041 every 8 connection(s)
I20260810 18:59:29.461956 10055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:59:29.462589 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 10055
I20260810 18:59:29.462702 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:59:29.463075 28238 external_mini_cluster.cc:1428] Setting key c4aef09b441894832327fb18e4750e02
I20260810 18:59:29.465163 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35677
--builtin_ntp_servers=127.27.147.148:34717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 18:59:29.472972 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:29.464070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:44747 (local address 127.27.147.190:35677)
0810 18:59:29.464217 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:29.464221 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:29.465439 (+ 1218us) server_negotiation.cc:407] Connection header received
0810 18:59:29.466957 (+ 1518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:29.466963 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:29.467018 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:29.467093 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:29.468331 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.468890 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.470801 (+ 1911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.471084 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.471164 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:29.472493 (+ 1329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:29.472496 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:29.472498 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:29.472510 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:29.472528 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:29.472537 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:29.472597 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:29.472804 (+ 207us) server_negotiation.cc:299] Negotiation successful
0810 18:59:29.472858 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:59:29.474341 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:29.464542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35677 (local address 127.27.147.130:44747)
0810 18:59:29.465294 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:29.465346 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:29.466656 (+ 1310us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:29.467312 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:29.467323 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:29.467439 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:29.468148 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:29.468175 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.469050 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.469057 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:29.470659 (+ 1602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:29.470676 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.471244 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.471253 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:29.471511 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:29.472159 (+ 648us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:59:29.472163 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:29.472223 (+ 60us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:59:29.472319 (+ 96us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:29.472630 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:29.472641 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:29.472649 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:59:29.472791 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 18:59:29.473214 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":147,"threads_started":1}
I20260810 18:59:29.475567 10248 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35677
I20260810 18:59:29.475821 10248 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:29.476330 10248 heartbeater.cc:511] Master 127.27.147.190:35677 requested a full tablet report, sending...
I20260810 18:59:29.477411 9749 ts_manager.cc:194] Registered new tserver with Master: ff1857b5c8bd472e9ce85b5f2dc4291b (127.27.147.130:38041)
I20260810 18:59:29.477993 9749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.130:44747
W20260810 18:59:29.583124 10254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:29.583379 10254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:29.583417 10254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:29.587736 10254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:29.587786 10254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:29.587867 10254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:29.592797 10254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35677
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:29.593816 10254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:29.595273 10254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:29.605084 10268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:29.605110 10264 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:29.605284 10254 server_base.cc:1034] running on GCE node
W20260810 18:59:29.605106 10263 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:29.605792 10254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:29.606398 10254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:29.607553 10254 hybrid_clock.cc:648] HybridClock initialized: now 1786388369607523 us; error 41 us; skew 500 ppm
I20260810 18:59:29.609447 10254 webserver.cc:460] Webserver started at http://127.27.147.131:43923/ using document root <none> and password file <none>
I20260810 18:59:29.610085 10254 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:29.610152 10254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:29.610394 10254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:29.612452 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "74712bec6026463e92d6f228b24f21ba"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "8cb7df4c429202f6b46cdc98da67c7d5"
server_key_iv: "46f5eb7d14ed65f29849fb38c7df26fd"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.613018 10254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "74712bec6026463e92d6f228b24f21ba"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "8cb7df4c429202f6b46cdc98da67c7d5"
server_key_iv: "46f5eb7d14ed65f29849fb38c7df26fd"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.616595 10254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 18:59:29.619287 10276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.620486 10254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:29.620621 10254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "74712bec6026463e92d6f228b24f21ba"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "8cb7df4c429202f6b46cdc98da67c7d5"
server_key_iv: "46f5eb7d14ed65f29849fb38c7df26fd"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.620730 10254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:29.641116 10254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:29.641845 10254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:29.642055 10254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:29.642653 10254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:29.643720 10254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:29.643796 10254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.643911 10254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:29.643955 10254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.652616 10254 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:42301
I20260810 18:59:29.652631 10468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:42301 every 8 connection(s)
I20260810 18:59:29.653532 10254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:29.662932 9807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:29.655485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:51649 (local address 127.27.147.190:35677)
0810 18:59:29.655619 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:29.655623 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:29.656649 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 18:59:29.658071 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:29.658075 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:29.658144 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:29.658237 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:29.659217 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.659761 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.660964 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.661207 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.661300 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:29.662424 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:29.662427 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:29.662429 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:29.662441 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:29.662458 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:29.662466 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:29.662528 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:29.662671 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 18:59:29.662744 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:59:29.663007 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 10254
I20260810 18:59:29.663101 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:59:29.663409 28238 external_mini_cluster.cc:1428] Setting key a69df56668b828dc9e46f6b2f04dedff
I20260810 18:59:29.663841 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:29.655823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35677 (local address 127.27.147.131:51649)
0810 18:59:29.656494 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:29.656534 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:29.657846 (+ 1312us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:29.658420 (+ 574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:29.658426 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:29.658508 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:29.659073 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:29.659089 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.659892 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.659896 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:29.660821 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:29.660831 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:29.661350 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:29.661353 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:29.661514 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:29.662187 (+ 673us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:59:29.662188 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:29.662228 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:59:29.662288 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:29.662544 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:29.662551 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:29.662555 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:59:29.662650 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 18:59:29.662940 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":108,"threads_started":1}
I20260810 18:59:29.665052 10472 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35677
I20260810 18:59:29.665304 10472 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:29.665889 10472 heartbeater.cc:511] Master 127.27.147.190:35677 requested a full tablet report, sending...
I20260810 18:59:29.666982 9749 ts_manager.cc:194] Registered new tserver with Master: 74712bec6026463e92d6f228b24f21ba (127.27.147.131:42301)
I20260810 18:59:29.667550 9749 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.131:51649
I20260810 18:59:29.677552 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:59:29.680434 9807 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:29.679551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.27.147.190:35677)
0810 18:59:29.679692 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:29.679696 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:29.679876 (+ 180us) server_negotiation.cc:407] Connection header received
0810 18:59:29.679976 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:29.679978 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:29.680014 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:29.680077 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:29.680307 (+ 230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50194: BlockingRecv error: recv got EOF from 127.0.0.1:50194 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 18:59:29.680740 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 9845
I20260810 18:59:29.699929 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 10055
I20260810 18:59:29.729328 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 10254
I20260810 18:59:29.740613 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 9701
2026-08-10T18:59:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (965 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T18:59:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:59:29Z Disabled control of system clock
I20260810 18:59:29.776644 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41973
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:41973
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:59:29.888293 10487 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:29.888548 10487 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:29.888584 10487 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:29.892567 10487 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:29.892620 10487 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:29.892637 10487 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:29.892653 10487 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:29.892668 10487 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:29.897307 10487 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:41973
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41973
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10487
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:29.898291 10487 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:29.899616 10487 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:29.908190 10495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:29.908234 10497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:29.908317 10487 server_base.cc:1034] running on GCE node
W20260810 18:59:29.908231 10503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:29.908977 10487 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:29.909977 10487 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:29.911175 10487 hybrid_clock.cc:648] HybridClock initialized: now 1786388369911158 us; error 37 us; skew 500 ppm
I20260810 18:59:29.913003 10487 webserver.cc:460] Webserver started at http://127.27.147.190:46663/ using document root <none> and password file <none>
I20260810 18:59:29.913590 10487 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:29.913686 10487 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:29.913925 10487 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:29.916010 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "cdde6b3504a643afb1b83457681c7b64"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "a2d9ef9c2a2b706c33986b37d8de5808"
server_key_iv: "c2f0f478eee5d1c6b3d07ecb281b12b2"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.916586 10487 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "cdde6b3504a643afb1b83457681c7b64"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "a2d9ef9c2a2b706c33986b37d8de5808"
server_key_iv: "c2f0f478eee5d1c6b3d07ecb281b12b2"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.920167 10487 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 18:59:29.922508 10512 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:29.923741 10487 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:29.923882 10487 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "cdde6b3504a643afb1b83457681c7b64"
format_stamp: "Formatted at 2026-08-10 18:59:29 on dist-test-slave-znh6"
server_key: "a2d9ef9c2a2b706c33986b37d8de5808"
server_key_iv: "c2f0f478eee5d1c6b3d07ecb281b12b2"
server_key_version: "encryptionkey@0"
I20260810 18:59:29.923978 10487 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:29.928411 10487 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:29.928978 10487 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:29.929152 10487 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:29.935925 10487 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:41973
I20260810 18:59:29.935931 10593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:41973 every 8 connection(s)
I20260810 18:59:29.936794 10487 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:59:29.939971 10594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:29.942426 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 10487
I20260810 18:59:29.942546 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:59:29.942870 28238 external_mini_cluster.cc:1428] Setting key 88f3c5b600015a4619b2411df2f47222
I20260810 18:59:29.945957 10594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64: Bootstrap starting.
I20260810 18:59:29.948253 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:29.944264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32806 (local address 127.27.147.190:41973)
0810 18:59:29.944926 (+ 662us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:29.944941 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:29.944974 (+ 33us) server_negotiation.cc:407] Connection header received
0810 18:59:29.945562 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:29.945577 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:29.945905 (+ 328us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:29.946220 (+ 315us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:29.947029 (+ 809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:29.947033 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:29.947045 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:29.947056 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:29.947110 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:29.947125 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:29.947228 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:29.947360 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 18:59:29.947616 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":151,"threads_started":1}
I20260810 18:59:29.948535 10594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:29.949270 10594 log.cc:824] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:29.951081 10594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64: No bootstrap required, opened a new log
I20260810 18:59:29.954313 10594 raft_consensus.cc:348] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde6b3504a643afb1b83457681c7b64" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41973 } }
I20260810 18:59:29.954533 10594 raft_consensus.cc:374] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:29.954623 10594 raft_consensus.cc:729] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cdde6b3504a643afb1b83457681c7b64, State: Initialized, Role: FOLLOWER
I20260810 18:59:29.955072 10594 consensus_queue.cc:260] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde6b3504a643afb1b83457681c7b64" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41973 } }
I20260810 18:59:29.955216 10594 raft_consensus.cc:388] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:59:29.955307 10594 raft_consensus.cc:482] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:59:29.955402 10594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:29.956203 10594 raft_consensus.cc:504] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde6b3504a643afb1b83457681c7b64" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41973 } }
I20260810 18:59:29.956526 10594 leader_election.cc:302] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cdde6b3504a643afb1b83457681c7b64; no voters:
I20260810 18:59:29.956815 10594 leader_election.cc:288] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:59:29.956939 10601 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:29.957216 10601 raft_consensus.cc:686] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [term 1 LEADER]: Becoming Leader. State: Replica: cdde6b3504a643afb1b83457681c7b64, State: Running, Role: LEADER
I20260810 18:59:29.957535 10601 consensus_queue.cc:237] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde6b3504a643afb1b83457681c7b64" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41973 } }
I20260810 18:59:29.957960 10594 sys_catalog.cc:564] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:29.959156 10604 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cdde6b3504a643afb1b83457681c7b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde6b3504a643afb1b83457681c7b64" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41973 } } }
I20260810 18:59:29.959195 10605 sys_catalog.cc:455] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cdde6b3504a643afb1b83457681c7b64. Latest consensus state: current_term: 1 leader_uuid: "cdde6b3504a643afb1b83457681c7b64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cdde6b3504a643afb1b83457681c7b64" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41973 } } }
I20260810 18:59:29.959307 10604 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:29.959324 10605 sys_catalog.cc:458] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:29.959738 10615 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:59:29.962674 10615 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:59:29.967823 10615 catalog_manager.cc:1347] Generated new cluster ID: 49e672735f9e4856811aa790a7a7dd2a
I20260810 18:59:29.967893 10615 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:59:29.980326 10615 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:59:29.980618 10615 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:59:29.987972 10615 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cdde6b3504a643afb1b83457681c7b64: Generated new TSK 0
I20260810 18:59:29.988601 10615 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 18:59:30.001847 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41973
--builtin_ntp_servers=127.27.147.148:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 18:59:30.117205 10631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:30.117568 10631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:30.117666 10631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:30.124415 10631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:30.124466 10631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:30.124540 10631 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:59:30.132237 10631 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:30.133574 10631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:30.135110 10631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:30.147140 10638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:30.147166 10641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:30.147158 10639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:30.152343 10631 server_base.cc:1034] running on GCE node
I20260810 18:59:30.152897 10631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:30.153729 10631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:30.154953 10631 hybrid_clock.cc:648] HybridClock initialized: now 1786388370154927 us; error 46 us; skew 500 ppm
I20260810 18:59:30.157053 10631 webserver.cc:460] Webserver started at http://127.27.147.129:33125/ using document root <none> and password file <none>
I20260810 18:59:30.157733 10631 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:30.157825 10631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:30.158064 10631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:30.160508 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "c92b9c1bf44b4ec9bbbd855217014a0e"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "d7059eb1b5e43f137fdbe0414eff63e2"
server_key_iv: "84494d98e3c3c3a732d65f057284c3a8"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.161092 10631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "c92b9c1bf44b4ec9bbbd855217014a0e"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "d7059eb1b5e43f137fdbe0414eff63e2"
server_key_iv: "84494d98e3c3c3a732d65f057284c3a8"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.164870 10631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 18:59:30.167379 10649 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.168582 10631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 18:59:30.168717 10631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "c92b9c1bf44b4ec9bbbd855217014a0e"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "d7059eb1b5e43f137fdbe0414eff63e2"
server_key_iv: "84494d98e3c3c3a732d65f057284c3a8"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.168828 10631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:30.173584 10631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:30.174183 10631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:30.174376 10631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:30.174968 10631 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:30.175853 10631 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:30.175935 10631 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.176010 10631 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:30.176065 10631 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.185298 10631 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:44049
I20260810 18:59:30.185321 10835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:44049 every 8 connection(s)
I20260810 18:59:30.186124 10631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:59:30.192171 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 10631
I20260810 18:59:30.192209 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:30.187979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:50861 (local address 127.27.147.190:41973)
0810 18:59:30.188130 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:30.188135 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:30.189189 (+ 1054us) server_negotiation.cc:407] Connection header received
0810 18:59:30.190493 (+ 1304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:30.190495 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:30.190534 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:30.190613 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:30.191740 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:30.191743 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:30.191745 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:30.191756 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:30.191775 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:30.191783 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:30.191857 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:30.192007 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 18:59:30.192061 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:59:30.192286 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 18:59:30.192556 28238 external_mini_cluster.cc:1428] Setting key fd2fb49b9fce153955f1ca6b64d549c8
I20260810 18:59:30.193084 10838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:30.188394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41973 (local address 127.27.147.129:50861)
0810 18:59:30.189042 (+ 648us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:30.189079 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:30.190214 (+ 1135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:30.190768 (+ 554us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:30.190774 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:30.190916 (+ 142us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:30.191498 (+ 582us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:59:30.191500 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:30.191537 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:59:30.191602 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:30.191859 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:30.191866 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:30.191870 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 18:59:30.192000 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 18:59:30.192173 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261,"thread_start_us":115,"threads_started":1}
I20260810 18:59:30.194314 10836 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41973
I20260810 18:59:30.194553 10836 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:30.194713 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41973
--builtin_ntp_servers=127.27.147.148:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 18:59:30.195031 10836 heartbeater.cc:511] Master 127.27.147.190:41973 requested a full tablet report, sending...
I20260810 18:59:30.196746 10534 ts_manager.cc:194] Registered new tserver with Master: c92b9c1bf44b4ec9bbbd855217014a0e (127.27.147.129:44049)
W20260810 18:59:30.305114 10839 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:30.305330 10839 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:30.305367 10839 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:30.309249 10839 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:30.309294 10839 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:30.309346 10839 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:59:30.313918 10839 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10839
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:30.314798 10839 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:30.316077 10839 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:30.325590 10851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:30.325646 10854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:30.325650 10850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:30.325800 10839 server_base.cc:1034] running on GCE node
I20260810 18:59:30.326408 10839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:30.327145 10839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:30.328336 10839 hybrid_clock.cc:648] HybridClock initialized: now 1786388370328311 us; error 43 us; skew 500 ppm
I20260810 18:59:30.330036 10839 webserver.cc:460] Webserver started at http://127.27.147.130:42789/ using document root <none> and password file <none>
I20260810 18:59:30.330603 10839 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:30.330698 10839 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:30.330981 10839 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:30.332931 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "f696cf2646df4dbdb6b07670d89967ed"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "d4027a3cbc0c533712e204c8d365cc91"
server_key_iv: "4c3c6a15976c9aad07c20ec7d24f8b30"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.333509 10839 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "f696cf2646df4dbdb6b07670d89967ed"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "d4027a3cbc0c533712e204c8d365cc91"
server_key_iv: "4c3c6a15976c9aad07c20ec7d24f8b30"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.337126 10839 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.003s
I20260810 18:59:30.339438 10865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.340509 10839 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:30.340642 10839 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "f696cf2646df4dbdb6b07670d89967ed"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "d4027a3cbc0c533712e204c8d365cc91"
server_key_iv: "4c3c6a15976c9aad07c20ec7d24f8b30"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.340751 10839 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:30.345042 10839 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:30.345599 10839 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:30.345772 10839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:30.346299 10839 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:30.347208 10839 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:30.347275 10839 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.347344 10839 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:30.347386 10839 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.355892 10839 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:45485
I20260810 18:59:30.355911 11043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:45485 every 8 connection(s)
I20260810 18:59:30.356675 10839 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:59:30.362166 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 10839
I20260810 18:59:30.362299 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:59:30.362622 28238 external_mini_cluster.cc:1428] Setting key fe2850169626791d38c82ee2f94fe6bb
I20260810 18:59:30.362879 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:30.358387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:60843 (local address 127.27.147.190:41973)
0810 18:59:30.358529 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:30.358533 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:30.359536 (+ 1003us) server_negotiation.cc:407] Connection header received
0810 18:59:30.360949 (+ 1413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:30.360952 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:30.361010 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:30.361090 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:30.362289 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:30.362295 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:30.362302 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:30.362314 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:30.362354 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:30.362367 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:30.362457 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:30.362618 (+ 161us) server_negotiation.cc:299] Negotiation successful
0810 18:59:30.362656 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 18:59:30.363819 11046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:30.358679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41973 (local address 127.27.147.130:60843)
0810 18:59:30.359386 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:30.359423 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:30.360740 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:30.361244 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:30.361252 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:30.361332 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:30.361931 (+ 599us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:59:30.361934 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:30.361989 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:59:30.362081 (+ 92us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:30.362454 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:30.362473 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:30.362481 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:59:30.362660 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 18:59:30.362924 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":137,"threads_started":1}
I20260810 18:59:30.364868 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41973
--builtin_ntp_servers=127.27.147.148:39457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 18:59:30.364904 11044 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41973
I20260810 18:59:30.365141 11044 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:30.365515 11044 heartbeater.cc:511] Master 127.27.147.190:41973 requested a full tablet report, sending...
I20260810 18:59:30.366592 10534 ts_manager.cc:194] Registered new tserver with Master: f696cf2646df4dbdb6b07670d89967ed (127.27.147.130:45485)
W20260810 18:59:30.482154 11047 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:30.482404 11047 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:30.482475 11047 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:30.486794 11047 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:30.486881 11047 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:30.486974 11047 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:30.492063 11047 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:39457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41973
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11047
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:30.493000 11047 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:30.494273 11047 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:30.503666 11057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:30.503832 11053 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:30.503842 11052 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:30.503933 11047 server_base.cc:1034] running on GCE node
I20260810 18:59:30.504415 11047 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:30.505038 11047 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:30.506253 11047 hybrid_clock.cc:648] HybridClock initialized: now 1786388370506225 us; error 41 us; skew 500 ppm
I20260810 18:59:30.508092 11047 webserver.cc:460] Webserver started at http://127.27.147.131:45519/ using document root <none> and password file <none>
I20260810 18:59:30.508656 11047 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:30.508749 11047 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:30.508984 11047 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:30.510979 11047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "cba70f9e17c540af95e6b92935699525"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "b032c6dc2052f7814bab7fa131f12469"
server_key_iv: "2c690cf797d7ee859dc203c835677a31"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.511528 11047 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "cba70f9e17c540af95e6b92935699525"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "b032c6dc2052f7814bab7fa131f12469"
server_key_iv: "2c690cf797d7ee859dc203c835677a31"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.515130 11047 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:59:30.517499 11072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.518599 11047 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 18:59:30.518743 11047 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "cba70f9e17c540af95e6b92935699525"
format_stamp: "Formatted at 2026-08-10 18:59:30 on dist-test-slave-znh6"
server_key: "b032c6dc2052f7814bab7fa131f12469"
server_key_iv: "2c690cf797d7ee859dc203c835677a31"
server_key_version: "encryptionkey@0"
I20260810 18:59:30.518875 11047 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:30.523371 11047 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:30.523939 11047 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:30.524113 11047 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:30.524637 11047 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:30.525511 11047 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:30.525583 11047 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.525658 11047 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:30.525702 11047 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:30.534451 11047 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:42049
I20260810 18:59:30.534482 11272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:42049 every 8 connection(s)
I20260810 18:59:30.535269 11047 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:30.540983 10599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:30.536947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:34915 (local address 127.27.147.190:41973)
0810 18:59:30.537078 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:30.537081 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:30.538007 (+ 926us) server_negotiation.cc:407] Connection header received
0810 18:59:30.539375 (+ 1368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:30.539379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:30.539439 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:30.539537 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:30.540511 (+ 974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:30.540514 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:30.540517 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 18:59:30.540530 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:30.540553 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 18:59:30.540567 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:30.540658 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:30.540811 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 18:59:30.540854 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:59:30.541725 11275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:30.537247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41973 (local address 127.27.147.131:34915)
0810 18:59:30.537860 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:30.537906 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:30.539160 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:30.539693 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:30.539700 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:30.539781 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:30.540267 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 18:59:30.540269 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:30.540305 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 18:59:30.540368 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:30.540657 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:30.540668 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:30.540676 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 18:59:30.540840 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 18:59:30.541076 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"thread_start_us":122,"threads_started":1}
I20260810 18:59:30.542785 11273 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41973
I20260810 18:59:30.543011 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 11047
I20260810 18:59:30.543062 11273 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:30.543138 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:59:30.543444 28238 external_mini_cluster.cc:1428] Setting key 9a18ecf60a78ddab6181558b1bdb0e43
I20260810 18:59:30.543547 11273 heartbeater.cc:511] Master 127.27.147.190:41973 requested a full tablet report, sending...
I20260810 18:59:30.544754 10534 ts_manager.cc:194] Registered new tserver with Master: cba70f9e17c540af95e6b92935699525 (127.27.147.131:42049)
I20260810 18:59:30.545444 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 18:59:30.548295 10599 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 18:59:30.547374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32820 (local address 127.27.147.190:41973)
0810 18:59:30.547520 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:30.547524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:30.547695 (+ 171us) server_negotiation.cc:407] Connection header received
0810 18:59:30.547842 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:30.547845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:30.547874 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:30.547959 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:30.548193 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:32820: BlockingRecv error: recv got EOF from 127.0.0.1:32820 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 18:59:30.548486 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 10631
I20260810 18:59:30.572935 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 10839
I20260810 18:59:30.586325 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 11047
I20260810 18:59:30.594174 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 10487
2026-08-10T18:59:30Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (854 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:59:30 dist-test-slave-znh6 krb5kdc[11287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:59:30 dist-test-slave-znh6 krb5kdc[11287](info): set up 2 sockets
Aug 10 18:59:30 dist-test-slave-znh6 krb5kdc[11287](info): commencing operation
krb5kdc: starting...
W20260810 18:59:31.856124 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.234s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:59:31 dist-test-slave-znh6 krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:59:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:59:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 18:59:31.981644 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34601
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:34601
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:32.092417 11310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:32.092653 11310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:32.092724 11310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:32.096797 11310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 18:59:32.096853 11310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:32.096873 11310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:32.096889 11310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 18:59:32.096940 11310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 18:59:32.101976 11310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:34601
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:32.102959 11310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:32.103953 11310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:32.112201 11318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:32.112236 11320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:32.112371 11310 server_base.cc:1034] running on GCE node
W20260810 18:59:32.112236 11322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:32.113066 11310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:32.114089 11310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:32.115245 11310 hybrid_clock.cc:648] HybridClock initialized: now 1786388372115220 us; error 38 us; skew 500 ppm
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:32.117882 11310 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 18:59:32.118762 11310 webserver.cc:460] Webserver started at http://127.27.147.190:44667/ using document root <none> and password file <none>
I20260810 18:59:32.119343 11310 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:32.119437 11310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:32.119673 11310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:32.121623 11310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "7e989947419f4ce083d64b38d3dfcb49"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "473851d4c2bf02f7643a530949830c2d"
server_key_iv: "ada72e24762973bf051991680a95866f"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.122195 11310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "7e989947419f4ce083d64b38d3dfcb49"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "473851d4c2bf02f7643a530949830c2d"
server_key_iv: "ada72e24762973bf051991680a95866f"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.125898 11310 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.003s
I20260810 18:59:32.128255 11335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.129451 11310 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 18:59:32.129591 11310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "7e989947419f4ce083d64b38d3dfcb49"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "473851d4c2bf02f7643a530949830c2d"
server_key_iv: "ada72e24762973bf051991680a95866f"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.129701 11310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:32.140758 11310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:32.143381 11310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:32.143584 11310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:32.150507 11310 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:34601
I20260810 18:59:32.150790 11416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:34601 every 8 connection(s)
I20260810 18:59:32.151494 11310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 18:59:32.154121 11419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:32.157222 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 11310
I20260810 18:59:32.157344 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 18:59:32.157655 28238 external_mini_cluster.cc:1428] Setting key 6d127bfee89528dd4e10792363a92607
I20260810 18:59:32.160033 11419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49: Bootstrap starting.
I20260810 18:59:32.162516 11419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:32.163481 11419 log.cc:824] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:32.165637 11419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49: No bootstrap required, opened a new log
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388371, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:32.169210 11419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e989947419f4ce083d64b38d3dfcb49" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34601 } }
I20260810 18:59:32.169461 11419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:32.169541 11419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e989947419f4ce083d64b38d3dfcb49, State: Initialized, Role: FOLLOWER
I20260810 18:59:32.169958 11419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e989947419f4ce083d64b38d3dfcb49" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34601 } }
I20260810 18:59:32.170109 11419 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 18:59:32.170198 11419 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 18:59:32.170320 11419 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:32.171213 11419 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e989947419f4ce083d64b38d3dfcb49" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34601 } }
I20260810 18:59:32.171561 11419 leader_election.cc:302] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e989947419f4ce083d64b38d3dfcb49; no voters:
I20260810 18:59:32.171852 11419 leader_election.cc:288] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 18:59:32.171958 11426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:32.172194 11426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [term 1 LEADER]: Becoming Leader. State: Replica: 7e989947419f4ce083d64b38d3dfcb49, State: Running, Role: LEADER
I20260810 18:59:32.172557 11426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e989947419f4ce083d64b38d3dfcb49" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34601 } }
I20260810 18:59:32.172626 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.159084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43756 (local address 127.27.147.190:34601)
0810 18:59:32.159517 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.159533 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.159568 (+ 35us) server_negotiation.cc:407] Connection header received
0810 18:59:32.160175 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.160197 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.160531 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.160796 (+ 265us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:32.161737 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.162784 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.164005 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.164563 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.164687 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.167489 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:32.167494 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:32.167506 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:32.167532 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:32.169241 (+ 1709us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.169751 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.169757 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.169782 (+ 25us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.169921 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.170247 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.170250 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.170252 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.170730 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:32.171014 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.171264 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.171580 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":147,"threads_started":1}
I20260810 18:59:32.173151 11419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 18:59:32.173696 11431 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e989947419f4ce083d64b38d3dfcb49. Latest consensus state: current_term: 1 leader_uuid: "7e989947419f4ce083d64b38d3dfcb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e989947419f4ce083d64b38d3dfcb49" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34601 } } }
I20260810 18:59:32.173827 11431 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:32.174180 11429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e989947419f4ce083d64b38d3dfcb49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e989947419f4ce083d64b38d3dfcb49" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34601 } } }
I20260810 18:59:32.174261 11429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49 [sys.catalog]: This master's current role is: LEADER
I20260810 18:59:32.175073 11439 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 18:59:32.177870 11439 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 18:59:32.182994 11439 catalog_manager.cc:1347] Generated new cluster ID: 1fe5b63cfc9a44aca12a9164b8c2a0ba
I20260810 18:59:32.183063 11439 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 18:59:32.195931 11439 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 18:59:32.196792 11439 catalog_manager.cc:1504] Loading token signing keys...
I20260810 18:59:32.206488 11439 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49: Generated new TSK 0
I20260810 18:59:32.207185 11439 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 18:59:32.241221 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34601
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:32.353415 11461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:32.353650 11461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:32.353690 11461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:32.357729 11461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:32.357784 11461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:32.357846 11461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 18:59:32.362946 11461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:32.363921 11461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:32.364858 11461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:32.374308 11469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:32.374344 11470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:32.374367 11461 server_base.cc:1034] running on GCE node
W20260810 18:59:32.374320 11472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:32.375044 11461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:32.375666 11461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:32.376827 11461 hybrid_clock.cc:648] HybridClock initialized: now 1786388372376813 us; error 28 us; skew 500 ppm
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:32.379535 11461 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 18:59:32.380337 11461 webserver.cc:460] Webserver started at http://127.27.147.129:41273/ using document root <none> and password file <none>
I20260810 18:59:32.380885 11461 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:32.380973 11461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:32.381202 11461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:32.383237 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "4f1ce84743b9415aab1ccdb0601fc894"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "7805240a9cee5215d24cf6a905f07407"
server_key_iv: "4e759b02620a30cbbe81a0688eb373dd"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.383833 11461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "4f1ce84743b9415aab1ccdb0601fc894"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "7805240a9cee5215d24cf6a905f07407"
server_key_iv: "4e759b02620a30cbbe81a0688eb373dd"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.387468 11461 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 18:59:32.389784 11483 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.391041 11461 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 18:59:32.391166 11461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "4f1ce84743b9415aab1ccdb0601fc894"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "7805240a9cee5215d24cf6a905f07407"
server_key_iv: "4e759b02620a30cbbe81a0688eb373dd"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.391271 11461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:32.413087 11461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:32.416026 11461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:32.416240 11461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:32.416852 11461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:32.417723 11461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:32.417811 11461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.417907 11461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:32.417959 11461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.427253 11461 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:39883
I20260810 18:59:32.427271 11674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:39883 every 8 connection(s)
I20260810 18:59:32.428133 11461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 18:59:32.437484 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 11461
I20260810 18:59:32.437608 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:32.437940 28238 external_mini_cluster.cc:1428] Setting key 522f0e20b6c4783ff866dc832fda5e2d
I20260810 18:59:32.442041 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.430048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:54547 (local address 127.27.147.190:34601)
0810 18:59:32.430187 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.430191 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.431054 (+ 863us) server_negotiation.cc:407] Connection header received
0810 18:59:32.432483 (+ 1429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.432486 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.432545 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.432650 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:32.433817 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.434382 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.435607 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.435853 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.435969 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.438504 (+ 2535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:32.438509 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:32.438511 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:32.438540 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:32.440020 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.440549 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.440553 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.440555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.440636 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.440977 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.440982 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.440984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.441161 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:32.441271 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.441740 (+ 469us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.441861 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 18:59:32.442809 11679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.430393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34601 (local address 127.27.147.129:54547)
0810 18:59:32.430914 (+ 521us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:32.430959 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:32.432246 (+ 1287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:32.432860 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:32.432867 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:32.433199 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:32.433664 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.433676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.434516 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.434525 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.435474 (+ 949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.435485 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.436010 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.436015 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.436183 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.436680 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:32.436696 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:32.438314 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:32.440162 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:32.440168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:32.440181 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:32.440442 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:32.440733 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:32.440737 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:32.440739 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:32.440879 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:32.441271 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:32.441277 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:32.441515 (+ 238us) client_negotiation.cc:769] Sending connection context
0810 18:59:32.441697 (+ 182us) client_negotiation.cc:240] Negotiation successful
0810 18:59:32.442000 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":111,"threads_started":1}
I20260810 18:59:32.444048 11675 heartbeater.cc:347] Connected to a master server at 127.27.147.190:34601
I20260810 18:59:32.444289 11675 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:32.444767 11675 heartbeater.cc:511] Master 127.27.147.190:34601 requested a full tablet report, sending...
I20260810 18:59:32.446249 11361 ts_manager.cc:194] Registered new tserver with Master: 4f1ce84743b9415aab1ccdb0601fc894 (127.27.147.129:39883)
I20260810 18:59:32.447420 11361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:54547
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 18:59:32.465994 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34601
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:32.577598 11684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:32.577855 11684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:32.577937 11684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:32.581964 11684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:32.582017 11684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:32.582101 11684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 18:59:32.588579 11684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:32.589555 11684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:32.590529 11684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:32.600168 11697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:32.600164 11693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:32.600164 11694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:32.600273 11684 server_base.cc:1034] running on GCE node
I20260810 18:59:32.600832 11684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:32.601420 11684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:32.602612 11684 hybrid_clock.cc:648] HybridClock initialized: now 1786388372602604 us; error 28 us; skew 500 ppm
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:32.605347 11684 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 18:59:32.606194 11684 webserver.cc:460] Webserver started at http://127.27.147.130:35541/ using document root <none> and password file <none>
I20260810 18:59:32.606742 11684 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:32.606853 11684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:32.607089 11684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:32.609014 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "6695c151071d0a836d89395ada7bc337"
server_key_iv: "72dc7fb34d1547737698927f418ef572"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.609576 11684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "6695c151071d0a836d89395ada7bc337"
server_key_iv: "72dc7fb34d1547737698927f418ef572"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.613181 11684 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.005s
I20260810 18:59:32.615566 11708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.616667 11684 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 18:59:32.616802 11684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "6695c151071d0a836d89395ada7bc337"
server_key_iv: "72dc7fb34d1547737698927f418ef572"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.616914 11684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:32.632162 11684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:32.635362 11684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:32.635557 11684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:32.636121 11684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:32.636917 11684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:32.636987 11684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.637058 11684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:32.637110 11684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.646061 11684 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:35109
I20260810 18:59:32.646096 11891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:35109 every 8 connection(s)
I20260810 18:59:32.647006 11684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 18:59:32.652168 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 11684
I20260810 18:59:32.652309 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 18:59:32.652611 28238 external_mini_cluster.cc:1428] Setting key 4cbfeb7b2d3720a947a31370f051e91d
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:32.661654 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.648741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:38671 (local address 127.27.147.190:34601)
0810 18:59:32.648913 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.648918 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.649716 (+ 798us) server_negotiation.cc:407] Connection header received
0810 18:59:32.651095 (+ 1379us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.651098 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.651145 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.651218 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:32.652332 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.652979 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.654190 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.654418 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.654498 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.657674 (+ 3176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:32.657678 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:32.657680 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:32.657702 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:32.659393 (+ 1691us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.659943 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.659949 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.659954 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.660055 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.660401 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.660407 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.660410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.660660 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:32.660767 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.661265 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.661413 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 18:59:32.662246 11895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.649137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34601 (local address 127.27.147.130:38671)
0810 18:59:32.649567 (+ 430us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:32.649608 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:32.650909 (+ 1301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:32.651373 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:32.651379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:32.651701 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:32.652171 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.652184 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.653126 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.653129 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.654063 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.654075 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.654575 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.654582 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.654831 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.655437 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:32.655463 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:32.657499 (+ 2036us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:32.659511 (+ 2012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:32.659516 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:32.659527 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:32.659822 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:32.660153 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:32.660155 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:32.660157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:32.660299 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:32.660785 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:32.660789 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:32.661012 (+ 223us) client_negotiation.cc:769] Sending connection context
0810 18:59:32.661168 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 18:59:32.661415 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":108,"threads_started":1}
I20260810 18:59:32.663507 11892 heartbeater.cc:347] Connected to a master server at 127.27.147.190:34601
I20260810 18:59:32.663761 11892 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:32.664234 11892 heartbeater.cc:511] Master 127.27.147.190:34601 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 18:59:32.665341 11361 ts_manager.cc:194] Registered new tserver with Master: 0ad32e5d209d4d14a2a5eca8d9e3f6d8 (127.27.147.130:35109)
I20260810 18:59:32.665942 11361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:38671
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 18:59:32.681797 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34601
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 18:59:32.791615 11899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 18:59:32.791836 11899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 18:59:32.791877 11899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 18:59:32.795943 11899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 18:59:32.795991 11899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 18:59:32.796085 11899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 18:59:32.801222 11899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 18:59:32.802071 11899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 18:59:32.802996 11899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 18:59:32.812556 11908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:32.812597 11906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 18:59:32.812597 11905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 18:59:32.812770 11899 server_base.cc:1034] running on GCE node
I20260810 18:59:32.813244 11899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 18:59:32.813833 11899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 18:59:32.815047 11899 hybrid_clock.cc:648] HybridClock initialized: now 1786388372815019 us; error 45 us; skew 500 ppm
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 18:59:32.817788 11899 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 18:59:32.818619 11899 webserver.cc:460] Webserver started at http://127.27.147.131:44985/ using document root <none> and password file <none>
I20260810 18:59:32.819227 11899 fs_manager.cc:359] Metadata directory not provided
I20260810 18:59:32.819329 11899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 18:59:32.819557 11899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 18:59:32.821607 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "ed5453bc38704749976fbafcf3508743"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "cdd0d3b2325289d43f85ea3a4ddcf845"
server_key_iv: "0ca714e20b4f90c080e58c5ddba668c6"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.822155 11899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "ed5453bc38704749976fbafcf3508743"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "cdd0d3b2325289d43f85ea3a4ddcf845"
server_key_iv: "0ca714e20b4f90c080e58c5ddba668c6"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.825786 11899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 18:59:32.828174 11917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.829385 11899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 18:59:32.829535 11899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "ed5453bc38704749976fbafcf3508743"
format_stamp: "Formatted at 2026-08-10 18:59:32 on dist-test-slave-znh6"
server_key: "cdd0d3b2325289d43f85ea3a4ddcf845"
server_key_iv: "0ca714e20b4f90c080e58c5ddba668c6"
server_key_version: "encryptionkey@0"
I20260810 18:59:32.829643 11899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 18:59:32.844573 11899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 18:59:32.847330 11899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 18:59:32.847522 11899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 18:59:32.848093 11899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 18:59:32.848914 11899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 18:59:32.848989 11899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.849061 11899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 18:59:32.849114 11899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 18:59:32.857991 11899 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:35889
I20260810 18:59:32.858018 12095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:35889 every 8 connection(s)
I20260810 18:59:32.858929 11899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 18:59:32.867609 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 11899
Aug 10 18:59:32 dist-test-slave-znh6 krb5kdc[11287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388372, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 18:59:32.867754 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 18:59:32.868046 28238 external_mini_cluster.cc:1428] Setting key e7faf9981878a3fe15afc01067f6d26f
I20260810 18:59:32.871925 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.860696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:53275 (local address 127.27.147.190:34601)
0810 18:59:32.860887 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.860892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.861903 (+ 1011us) server_negotiation.cc:407] Connection header received
0810 18:59:32.862719 (+ 816us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.862721 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.862766 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.862865 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:32.863984 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.864504 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.865600 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.865811 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.865893 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.868335 (+ 2442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:32.868361 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:32.868366 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:32.868404 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:32.870004 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.870587 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.870590 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.870591 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.870665 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.871040 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.871042 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.871043 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.871211 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:32.871287 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.871713 (+ 426us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.871808 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 18:59:32.872711 12099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.861097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34601 (local address 127.27.147.131:53275)
0810 18:59:32.861744 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:32.861788 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:32.862542 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:32.863023 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:32.863029 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:32.863407 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0810 18:59:32.863859 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.863873 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.864621 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.864624 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.865489 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.865498 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.865940 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.865943 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.866080 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.866561 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 18:59:32.866576 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 18:59:32.868151 (+ 1575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 18:59:32.870176 (+ 2025us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:32.870183 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:32.870197 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:32.870496 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:32.870748 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 18:59:32.870750 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 18:59:32.870751 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 18:59:32.870957 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 18:59:32.871302 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 18:59:32.871306 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 18:59:32.871516 (+ 210us) client_negotiation.cc:769] Sending connection context
0810 18:59:32.871664 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 18:59:32.871924 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":120,"threads_started":1}
I20260810 18:59:32.873826 12097 heartbeater.cc:347] Connected to a master server at 127.27.147.190:34601
I20260810 18:59:32.874076 12097 heartbeater.cc:464] Registering TS with master...
I20260810 18:59:32.874672 12097 heartbeater.cc:511] Master 127.27.147.190:34601 requested a full tablet report, sending...
I20260810 18:59:32.875734 11361 ts_manager.cc:194] Registered new tserver with Master: ed5453bc38704749976fbafcf3508743 (127.27.147.131:35889)
I20260810 18:59:32.876313 11361 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:53275
I20260810 18:59:32.881778 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 18:59:32.890861 11423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.883733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43772 (local address 127.27.147.190:34601)
0810 18:59:32.883860 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.883863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.884105 (+ 242us) server_negotiation.cc:407] Connection header received
0810 18:59:32.884256 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.884259 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.884300 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.884394 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 18:59:32.885077 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.885581 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.886427 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.886651 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.886758 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.888023 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 18:59:32.888027 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 18:59:32.888029 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 18:59:32.888054 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 18:59:32.889179 (+ 1125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.889600 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.889605 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.889607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.889691 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 18:59:32.889981 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 18:59:32.889984 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 18:59:32.889986 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 18:59:32.890193 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 18:59:32.890330 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.890564 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.890675 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 18:59:32.893604 11361 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43772:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 18:59:32.896903 11361 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 18:59:32.911343 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.905917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41366 (local address 127.27.147.131:35889)
0810 18:59:32.906248 (+ 331us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.906258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.906410 (+ 152us) server_negotiation.cc:407] Connection header received
0810 18:59:32.906529 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.906533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.906670 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.906987 (+ 317us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:32.907402 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.908182 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.909958 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.910592 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.910732 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.910885 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.911093 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.911184 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":231,"thread_start_us":133,"threads_started":1}
I20260810 18:59:32.911481 12113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.904866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:35109 (local address 127.0.0.1:42616)
0810 18:59:32.905569 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:32.905619 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:32.905737 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:32.907141 (+ 1404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:32.907145 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:32.907161 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:32.907298 (+ 137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.907305 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.908843 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.908846 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.910054 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.910063 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.911112 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.911116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.911202 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.911228 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 18:59:32.911269 (+ 41us) client_negotiation.cc:240] Negotiation successful
0810 18:59:32.911316 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":522,"mutex_wait_us":78,"thread_start_us":55,"threads_started":1}
I20260810 18:59:32.911338 12117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.905500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:35889 (local address 127.0.0.1:41366)
0810 18:59:32.906309 (+ 809us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:32.906323 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:32.906435 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:32.906861 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:32.906868 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:32.906895 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:32.907271 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.907281 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.908351 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.908357 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.909821 (+ 1464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.909833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.910721 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.910725 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.910880 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.910929 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 18:59:32.911061 (+ 132us) client_negotiation.cc:240] Negotiation successful
0810 18:59:32.911180 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":752,"mutex_wait_us":86,"thread_start_us":95,"threads_started":1}
I20260810 18:59:32.911922 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.905067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:39883 (local address 127.0.0.1:39112)
0810 18:59:32.905983 (+ 916us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:32.905999 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:32.906098 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:32.907040 (+ 942us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:32.907046 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:32.907069 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:32.907299 (+ 230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.907305 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.908803 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.908808 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.909799 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:32.909809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.911012 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.911019 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:32.911131 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.911156 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 18:59:32.911222 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 18:59:32.911295 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"mutex_wait_us":80,"thread_start_us":101,"threads_started":1}
I20260810 18:59:32.912138 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.905618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39112 (local address 127.27.147.129:39883)
0810 18:59:32.906331 (+ 713us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.906350 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.906376 (+ 26us) server_negotiation.cc:407] Connection header received
0810 18:59:32.906459 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.906467 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.906677 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.906854 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:32.907439 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.908641 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.909924 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.910671 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.910794 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.910939 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.911415 (+ 476us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.911518 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":90,"threads_started":1}
I20260810 18:59:32.912580 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:32.905040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42616 (local address 127.27.147.130:35109)
0810 18:59:32.906688 (+ 1648us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:32.906701 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:32.906721 (+ 20us) server_negotiation.cc:407] Connection header received
0810 18:59:32.906798 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:32.906803 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:32.906989 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:32.907467 (+ 478us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:32.907622 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.908709 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.910171 (+ 1462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:32.910986 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:32.912166 (+ 1180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:32.912301 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:32.912352 (+ 51us) server_negotiation.cc:299] Negotiation successful
0810 18:59:32.912438 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1547,"thread_start_us":102,"threads_started":1}
I20260810 18:59:32.914769 11784 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed7748c282f4b56ac201b6ebae23f7e (DEFAULT_TABLE table=test-table [id=014442a5bb7c4b6fb58afc079a7285c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:32.914779 11988 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed7748c282f4b56ac201b6ebae23f7e (DEFAULT_TABLE table=test-table [id=014442a5bb7c4b6fb58afc079a7285c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:32.914781 11557 tablet_service.cc:1467] Processing CreateTablet for tablet 0ed7748c282f4b56ac201b6ebae23f7e (DEFAULT_TABLE table=test-table [id=014442a5bb7c4b6fb58afc079a7285c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 18:59:32.915789 11784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed7748c282f4b56ac201b6ebae23f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:32.915830 11988 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed7748c282f4b56ac201b6ebae23f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:32.915824 11557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0ed7748c282f4b56ac201b6ebae23f7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 18:59:32.922183 12121 tablet_bootstrap.cc:492] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Bootstrap starting.
I20260810 18:59:32.923642 12123 tablet_bootstrap.cc:492] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Bootstrap starting.
I20260810 18:59:32.923676 12124 tablet_bootstrap.cc:492] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894: Bootstrap starting.
I20260810 18:59:32.925073 12121 tablet_bootstrap.cc:654] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:32.925383 12124 tablet_bootstrap.cc:654] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:32.925788 12123 tablet_bootstrap.cc:654] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Neither blocks nor log segments found. Creating new log.
I20260810 18:59:32.925954 12121 log.cc:824] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:32.926615 12124 log.cc:824] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:32.927057 12123 log.cc:824] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Log is configured to *not* fsync() on all Append() calls
I20260810 18:59:32.928189 12121 tablet_bootstrap.cc:492] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: No bootstrap required, opened a new log
I20260810 18:59:32.928398 12124 tablet_bootstrap.cc:492] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894: No bootstrap required, opened a new log
I20260810 18:59:32.928390 12121 ts_tablet_manager.cc:1397] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 18:59:32.928591 12124 ts_tablet_manager.cc:1397] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 18:59:32.931377 12121 raft_consensus.cc:348] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:32.931370 12124 raft_consensus.cc:348] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:32.931603 12124 raft_consensus.cc:374] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:32.931604 12121 raft_consensus.cc:374] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:32.931670 12121 raft_consensus.cc:729] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ad32e5d209d4d14a2a5eca8d9e3f6d8, State: Initialized, Role: FOLLOWER
I20260810 18:59:32.931690 12124 raft_consensus.cc:729] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f1ce84743b9415aab1ccdb0601fc894, State: Initialized, Role: FOLLOWER
I20260810 18:59:32.932107 12121 consensus_queue.cc:260] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:32.932120 12124 consensus_queue.cc:260] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:32.932626 12123 tablet_bootstrap.cc:492] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: No bootstrap required, opened a new log
I20260810 18:59:32.932787 11892 heartbeater.cc:503] Master 127.27.147.190:34601 was elected leader, sending a full tablet report...
I20260810 18:59:32.932871 12123 ts_tablet_manager.cc:1397] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.005s
I20260810 18:59:32.933228 12124 ts_tablet_manager.cc:1428] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894: Time spent starting tablet: real 0.005s user 0.001s sys 0.004s
I20260810 18:59:32.933605 11675 heartbeater.cc:503] Master 127.27.147.190:34601 was elected leader, sending a full tablet report...
I20260810 18:59:32.934659 12121 ts_tablet_manager.cc:1428] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260810 18:59:32.937012 12123 raft_consensus.cc:348] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:32.937201 12123 raft_consensus.cc:374] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 18:59:32.937320 12123 raft_consensus.cc:729] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed5453bc38704749976fbafcf3508743, State: Initialized, Role: FOLLOWER
I20260810 18:59:32.937748 12123 consensus_queue.cc:260] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:32.938364 12097 heartbeater.cc:503] Master 127.27.147.190:34601 was elected leader, sending a full tablet report...
I20260810 18:59:32.938629 12123 ts_tablet_manager.cc:1428] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 18:59:33.054569 12134 raft_consensus.cc:482] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 18:59:33.054827 12134 raft_consensus.cc:504] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:33.056048 12134 leader_election.cc:288] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ad32e5d209d4d14a2a5eca8d9e3f6d8 (127.27.147.130:35109), ed5453bc38704749976fbafcf3508743 (127.27.147.131:35889)
I20260810 18:59:33.061395 12115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:33.056374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:53669 (local address 127.27.147.130:35109)
0810 18:59:33.056517 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:33.056523 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:33.056687 (+ 164us) server_negotiation.cc:407] Connection header received
0810 18:59:33.056841 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:33.056847 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:33.056904 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:33.056999 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:33.057420 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.058230 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.060035 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.060652 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.060781 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:33.060834 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:33.061192 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 18:59:33.061264 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 18:59:33.061394 12120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:33.056458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:57543 (local address 127.27.147.131:35889)
0810 18:59:33.056592 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:33.056597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:33.056807 (+ 210us) server_negotiation.cc:407] Connection header received
0810 18:59:33.056871 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:33.056873 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:33.056922 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:33.056999 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 18:59:33.057420 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.058214 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.060035 (+ 1821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.060652 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.060781 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:33.060840 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:33.061192 (+ 352us) server_negotiation.cc:299] Negotiation successful
0810 18:59:33.061263 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 18:59:33.061445 12142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:33.056366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:35889 (local address 127.27.147.129:57543)
0810 18:59:33.056690 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:33.056703 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:33.056791 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:33.057026 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:33.057029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:33.057073 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:33.057294 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:33.057302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.058337 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.058341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:33.059902 (+ 1561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:33.059909 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.060785 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.060788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:33.061012 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:33.061067 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 18:59:33.061145 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 18:59:33.061263 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"mutex_wait_us":278,"spinlock_wait_cycles":30336,"thread_start_us":127,"threads_started":1}
I20260810 18:59:33.061448 12141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:33.056214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:35109 (local address 127.27.147.129:53669)
0810 18:59:33.056576 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:33.056593 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 18:59:33.056711 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 18:59:33.057026 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 18:59:33.057029 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 18:59:33.057074 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 18:59:33.057294 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:33.057302 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.058337 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.058341 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:33.059895 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 18:59:33.059909 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.060785 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.060788 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 18:59:33.060995 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:33.061064 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 18:59:33.061145 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 18:59:33.061261 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":283,"mutex_wait_us":279,"spinlock_wait_cycles":16640,"thread_start_us":80,"threads_started":1}
I20260810 18:59:33.062249 12015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7748c282f4b56ac201b6ebae23f7e" candidate_uuid: "4f1ce84743b9415aab1ccdb0601fc894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed5453bc38704749976fbafcf3508743" is_pre_election: true
I20260810 18:59:33.062247 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7748c282f4b56ac201b6ebae23f7e" candidate_uuid: "4f1ce84743b9415aab1ccdb0601fc894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" is_pre_election: true
I20260810 18:59:33.062546 11811 raft_consensus.cc:2454] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f1ce84743b9415aab1ccdb0601fc894 in term 0.
I20260810 18:59:33.062561 12015 raft_consensus.cc:2454] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4f1ce84743b9415aab1ccdb0601fc894 in term 0.
I20260810 18:59:33.063087 11488 leader_election.cc:302] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 0ad32e5d209d4d14a2a5eca8d9e3f6d8, 4f1ce84743b9415aab1ccdb0601fc894, ed5453bc38704749976fbafcf3508743; no voters:
I20260810 18:59:33.063405 12134 raft_consensus.cc:2781] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 18:59:33.063498 12134 raft_consensus.cc:482] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 18:59:33.063544 12134 raft_consensus.cc:3037] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:33.064611 12134 raft_consensus.cc:504] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:33.064963 12134 leader_election.cc:288] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [CANDIDATE]: Term 1 election: Requested vote from peers 0ad32e5d209d4d14a2a5eca8d9e3f6d8 (127.27.147.130:35109), ed5453bc38704749976fbafcf3508743 (127.27.147.131:35889)
I20260810 18:59:33.065294 11811 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7748c282f4b56ac201b6ebae23f7e" candidate_uuid: "4f1ce84743b9415aab1ccdb0601fc894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8"
I20260810 18:59:33.065383 12015 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0ed7748c282f4b56ac201b6ebae23f7e" candidate_uuid: "4f1ce84743b9415aab1ccdb0601fc894" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed5453bc38704749976fbafcf3508743"
I20260810 18:59:33.065428 11811 raft_consensus.cc:3037] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:33.065498 12015 raft_consensus.cc:3037] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 0 FOLLOWER]: Advancing to term 1
I20260810 18:59:33.066749 11811 raft_consensus.cc:2454] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f1ce84743b9415aab1ccdb0601fc894 in term 1.
I20260810 18:59:33.066749 12015 raft_consensus.cc:2454] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4f1ce84743b9415aab1ccdb0601fc894 in term 1.
I20260810 18:59:33.067060 11488 leader_election.cc:302] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ad32e5d209d4d14a2a5eca8d9e3f6d8, 4f1ce84743b9415aab1ccdb0601fc894; no voters:
I20260810 18:59:33.067239 12134 raft_consensus.cc:2781] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 18:59:33.067471 12134 raft_consensus.cc:686] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 1 LEADER]: Becoming Leader. State: Replica: 4f1ce84743b9415aab1ccdb0601fc894, State: Running, Role: LEADER
I20260810 18:59:33.067817 12134 consensus_queue.cc:237] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } }
I20260810 18:59:33.071015 11361 catalog_manager.cc:5495] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4f1ce84743b9415aab1ccdb0601fc894 (127.27.147.129). New cstate: current_term: 1 leader_uuid: "4f1ce84743b9415aab1ccdb0601fc894" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f1ce84743b9415aab1ccdb0601fc894" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39883 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 } health_report { overall_health: UNKNOWN } } }
I20260810 18:59:33.097983 12119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:33.094288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39126 (local address 127.27.147.129:39883)
0810 18:59:33.094428 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 18:59:33.094433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 18:59:33.094497 (+ 64us) server_negotiation.cc:407] Connection header received
0810 18:59:33.094674 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 18:59:33.094677 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 18:59:33.094741 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 18:59:33.094865 (+ 124us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 18:59:33.095106 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.095625 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.096538 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 18:59:33.096769 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 18:59:33.096877 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 18:59:33.097210 (+ 333us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 18:59:33.097600 (+ 390us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 18:59:33.097692 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0810 18:59:33.097834 (+ 142us) server_negotiation.cc:299] Negotiation successful
0810 18:59:33.097883 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 18:59:33.103549 12015 raft_consensus.cc:1261] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 1 FOLLOWER]: Refusing update from remote peer 4f1ce84743b9415aab1ccdb0601fc894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:33.103574 11811 raft_consensus.cc:1261] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 1 FOLLOWER]: Refusing update from remote peer 4f1ce84743b9415aab1ccdb0601fc894: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 18:59:33.104264 12134 consensus_queue.cc:1035] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed5453bc38704749976fbafcf3508743" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 18:59:33.104511 12143 consensus_queue.cc:1035] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ad32e5d209d4d14a2a5eca8d9e3f6d8" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35109 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 18:59:33.109798 12098 tablet.cc:2369] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 18:59:33.113413 12148 mvcc.cc:205] Tried to move back new op lower bound from 7317046776225366016 to 7317046776088551424. Current Snapshot: MvccSnapshot[applied={T|T < 7317046776225366016}]
I20260810 18:59:33.113581 12147 mvcc.cc:205] Tried to move back new op lower bound from 7317046776225366016 to 7317046776088551424. Current Snapshot: MvccSnapshot[applied={T|T < 7317046776225366016}]
I20260810 18:59:33.118989 11361 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:33.119165 11361 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43772:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 18:59:33.121646 11361 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7e989947419f4ce083d64b38d3dfcb49: Sending DeleteTablet for 3 replicas of tablet 0ed7748c282f4b56ac201b6ebae23f7e
I20260810 18:59:33.122509 11784 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed7748c282f4b56ac201b6ebae23f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:33 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:42616
I20260810 18:59:33.122505 11988 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed7748c282f4b56ac201b6ebae23f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:33 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:41366
I20260810 18:59:33.122907 11557 tablet_service.cc:1514] Processing DeleteTablet for tablet 0ed7748c282f4b56ac201b6ebae23f7e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 18:59:33 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:39112
I20260810 18:59:33.122977 12158 tablet_replica.cc:330] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: stopping tablet replica
I20260810 18:59:33.123070 12159 tablet_replica.cc:330] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: stopping tablet replica
I20260810 18:59:33.123287 12158 raft_consensus.cc:2229] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:33.123286 12159 raft_consensus.cc:2229] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 18:59:33.123585 12159 raft_consensus.cc:2258] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:33.123636 12161 tablet_replica.cc:330] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894: stopping tablet replica
I20260810 18:59:33.123986 12161 raft_consensus.cc:2229] T 0ed7748c282f4b56ac201b6ebae23f7e P 4f1ce84743b9415aab1ccdb0601fc894 [term 1 LEADER]: Raft consensus shutting down.
I20260810 18:59:33.124009 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 11461
I20260810 18:59:33.127731 12159 ts_tablet_manager.cc:1905] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:33.131672 12159 ts_tablet_manager.cc:1918] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:33.131898 12159 log.cc:1196] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/0ed7748c282f4b56ac201b6ebae23f7e
I20260810 18:59:33.132519 12159 ts_tablet_manager.cc:1939] T 0ed7748c282f4b56ac201b6ebae23f7e P ed5453bc38704749976fbafcf3508743: Deleting consensus metadata
I20260810 18:59:33.136059 11339 catalog_manager.cc:4841] TS ed5453bc38704749976fbafcf3508743 (127.27.147.131:35889): tablet 0ed7748c282f4b56ac201b6ebae23f7e (table test-table [id=014442a5bb7c4b6fb58afc079a7285c1]) successfully deleted
I20260810 18:59:33.137526 12158 raft_consensus.cc:2258] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 18:59:33.138859 12158 ts_tablet_manager.cc:1905] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 18:59:33.140972 12158 ts_tablet_manager.cc:1918] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 18:59:33.141088 12158 log.cc:1196] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/0ed7748c282f4b56ac201b6ebae23f7e
I20260810 18:59:33.141436 12158 ts_tablet_manager.cc:1939] T 0ed7748c282f4b56ac201b6ebae23f7e P 0ad32e5d209d4d14a2a5eca8d9e3f6d8: Deleting consensus metadata
I20260810 18:59:33.142463 11345 catalog_manager.cc:4841] TS 0ad32e5d209d4d14a2a5eca8d9e3f6d8 (127.27.147.130:35109): tablet 0ed7748c282f4b56ac201b6ebae23f7e (table test-table [id=014442a5bb7c4b6fb58afc079a7285c1]) successfully deleted
W20260810 18:59:33.143823 11338 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:39883 (error 108)
I20260810 18:59:33.144586 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 11684
I20260810 18:59:33.144812 12116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 18:59:33.144520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:39883 (local address 127.0.0.1:39140)
0810 18:59:33.144645 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 18:59:33.144719 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:39883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 18:59:33.146272 11338 catalog_manager.cc:4584] TS 4f1ce84743b9415aab1ccdb0601fc894 (127.27.147.129:39883): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0ed7748c282f4b56ac201b6ebae23f7e: Network error: Client connection negotiation failed: client connection to 127.27.147.129:39883: connect: Connection refused (error 111)
I20260810 18:59:33.154132 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 11899
I20260810 18:59:33.163689 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 11310
2026-08-10T18:59:33Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2569 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 18:59:33 dist-test-slave-znh6 krb5kdc[12166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 18:59:33 dist-test-slave-znh6 krb5kdc[12166](info): set up 2 sockets
Aug 10 18:59:33 dist-test-slave-znh6 krb5kdc[12166](info): commencing operation
krb5kdc: starting...
W20260810 18:59:34.406921 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 18:59:34 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388374, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T18:59:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T18:59:34Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/rangeradmin_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/rangeradmin_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/rangerlookup_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/rangerlookup_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
I20260810 18:59:34.577375 28238 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfzwUfv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/postgres -l logfile start
2026-08-10 18:59:37.136 UTC [12214] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 18:59:37.136 UTC [12214] LOG: listening on IPv4 address "127.27.147.148", port 51659
2026-08-10 18:59:37.140 UTC [12214] LOG: listening on Unix socket "/tmp/.s.PGSQL.51659"
2026-08-10 18:59:37.175 UTC [12217] LOG: database system was shut down at 2026-08-10 18:59:35 UTC
2026-08-10 18:59:37.180 UTC [12214] LOG: database system is ready to accept connections
I20260810 18:59:38.317826 28238 mini_postgres.cc:96] Postgres bound to 51659
2026-08-10 18:59:38.325 UTC [12230] FATAL: database "slave" does not exist
127.27.147.148:51659 - accepting connections
I20260810 18:59:38.326480 28238 mini_ranger.cc:161] Starting Ranger...
I20260810 18:59:38.342272 28238 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 18:59:38.804698 28238 mini_ranger.cc:87] Created ranger Postgres database
I20260810 18:59:38.804781 28238 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-admin
2026-08-10 18:59:39,097 [I] DB FLAVOR :POSTGRES
2026-08-10 18:59:39,097 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 18:59:39,097 [I] Checking connection..
2026-08-10 18:59:39,097 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 18:59:39,341 [I] Checking connection passed.
2026-08-10 18:59:39,342 [I] --------- Verifying version history table ---------
2026-08-10 18:59:39,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:59:39,589 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:59:39,590 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:59:39,854 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 18:59:39,854 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 18:59:39,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 18:59:40.108 UTC [12348] WARNING: there is no transaction in progress
2026-08-10 18:59:40,115 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 18:59:40,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 18:59:40,370 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 18:59:40,370 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 18:59:40,370 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 18:59:40,619 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 18:59:40,882 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-znh6.c.gcp-upstream.internal','N') ;"
2026-08-10 18:59:41,126 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:59:41,366 [I] Table x_portal_user does not exist in database ranger
2026-08-10 18:59:41,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:59:41,634 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 18:59:41,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:59:41,894 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:59:42,149 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 18:59:42,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 18:59:43.105 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.121 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.136 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.151 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.166 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.188 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.255 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.262 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.275 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.285 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.295 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.309 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.318 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.324 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.330 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43.337 UTC [12520] WARNING: there is no transaction in progress
2026-08-10 18:59:43,983 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 18:59:43,983 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 18:59:44,220 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 18:59:44,221 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 18:59:44,467 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 18:59:44,468 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 18:59:44,724 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 18:59:44,724 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 18:59:44,977 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 18:59:45,234 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-znh6.c.gcp-upstream.internal';"
2026-08-10 18:59:45,474 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 18:59:45,474 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 18:59:45,718 [I] DB_PATCHES have already been applied
I20260810 18:59:45.724535 28238 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfzwUfv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 18:59:45.724597 28238 mini_ranger.cc:193] Using host: 127.27.147.148
I20260810 18:59:45.729243 28238 mini_ranger.cc:227] Ranger admin URL: http://127.27.147.148:38745
Aug 10, 2026 6:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 6:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 6:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 6:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 6:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 6:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 6:59:46 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 18:59:47.077607 28238 mini_ranger.cc:160] Time spent starting Ranger: real 8.751s user 0.000s sys 0.012s
Aug 10, 2026 7:00:07 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:00:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:00:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:00:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:00:08 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 19:00:09.054354 28238 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/rangerkms_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/rangerkms_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.147.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.147.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 19:00:09 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388409, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 19:00:09.153716 28238 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 19:00:09.164611 28238 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 19:00:10.050782 28238 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 19:00:10.050915 28238 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms
I20260810 19:00:10.051029 28238 mini_ranger_kms.cc:227] Using postgres at 127.27.147.148:51659
2026-08-10 19:00:10,602 [I] DB FLAVOR :POSTGRES
2026-08-10 19:00:10,602 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:00:10,602 [I] Checking connection
2026-08-10 19:00:10,602 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 19:00:10,864 [I] connection success
2026-08-10 19:00:10,865 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 19:00:10,865 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 19:00:10,865 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 19:00:11,103 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 19:00:11,103 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:00:11,103 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 19:00:11,103 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:51659/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 19:00:11,367 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 19:00:11.861800 28238 mini_ranger_kms.cc:310] Created kms service
I20260810 19:00:12.028719 28238 mini_ranger_kms.cc:326] Created kudu user
I20260810 19:00:12.074142 28238 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 19:00:12 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388409, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:00:12.475164 28238 mini_ranger_kms.cc:384] Added ranger policy
I20260810 19:00:12.475335 28238 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 19:00:12.475364 28238 mini_ranger_kms.cc:240] Using host: 127.27.147.148
I20260810 19:00:12.480486 28238 mini_ranger_kms.cc:276] Ranger KMS PID: 12807
I20260810 19:00:12.480643 28238 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.147.148:54249
Aug 10, 2026 7:00:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:00:13 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:00:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:00:13 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:00:14 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 19:00:15 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786388415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.147.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:00:15.338843 28238 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.185s user 0.015s sys 0.351s
I20260810 19:00:15.338994 28238 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 19:00:15.339130 28238 mini_ranger_kms.cc:401] 127.27.147.148:54249/kms/v1/keys
Aug 10 19:00:15 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.147.148@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
Aug 10 19:00:16 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388415, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.147.148@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
Aug 10 19:00:18 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:00:18.577741 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35935
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:35935
--ranger_config_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:54249/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:00:18.731473 12886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:00:18.731706 12886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:00:18.731812 12886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:00:18.736011 12886 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:00:18.736065 12886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:00:18.736114 12886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:00:18.736145 12886 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:00:18.736183 12886 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:00:18.741586 12886 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.147.148:54249/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:35935
--ranger_config_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12886
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:00:18.742522 12886 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:00:18.743592 12886 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:00:18.753021 12886 server_base.cc:1034] running on GCE node
W20260810 19:00:18.753075 12896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:00:18.753217 12894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:00:18.753130 12903 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:00:18.753782 12886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:00:18.754911 12886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:00:18.756078 12886 hybrid_clock.cc:648] HybridClock initialized: now 1786388418756060 us; error 32 us; skew 500 ppm
Aug 10 19:00:18 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:00:18.763105 12886 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:00:18.764019 12886 webserver.cc:460] Webserver started at http://127.27.147.190:38697/ using document root <none> and password file <none>
I20260810 19:00:18.764515 12886 fs_manager.cc:359] Metadata directory not provided
I20260810 19:00:18.764568 12886 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:00:18.764803 12886 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:00:18 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388418, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:00:18.833534 12886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "8a006f6de9234a9db8015aa589b3b749"
format_stamp: "Formatted at 2026-08-10 19:00:18 on dist-test-slave-znh6"
server_key: "e535ddb20cdc369526bd2a16c5a8ebe6"
server_key_iv: "3d43f6ee2d9874f9c71314eb39b2d7f2"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:18.834241 12886 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "8a006f6de9234a9db8015aa589b3b749"
format_stamp: "Formatted at 2026-08-10 19:00:18 on dist-test-slave-znh6"
server_key: "e535ddb20cdc369526bd2a16c5a8ebe6"
server_key_iv: "3d43f6ee2d9874f9c71314eb39b2d7f2"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:18.838613 12886 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 19:00:18.852998 12911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:18.854468 12886 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:00:18.854615 12886 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "8a006f6de9234a9db8015aa589b3b749"
format_stamp: "Formatted at 2026-08-10 19:00:18 on dist-test-slave-znh6"
server_key: "e535ddb20cdc369526bd2a16c5a8ebe6"
server_key_iv: "3d43f6ee2d9874f9c71314eb39b2d7f2"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:18.854760 12886 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:00:18.869571 12886 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:00:18.872790 12886 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:00:18.873063 12886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:00:18.890900 12886 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:35935
I20260810 19:00:18.890987 12985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:35935 every 8 connection(s)
I20260810 19:00:18.892105 12886 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:00:18.895495 12987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:00:18.896739 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 12886
I20260810 19:00:18.896883 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:00:18.897194 28238 external_mini_cluster.cc:1428] Setting key cf1ff79826f61cbf0c97003cef82c1cc
I20260810 19:00:18.904453 12987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749: Bootstrap starting.
I20260810 19:00:18.907835 12987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749: Neither blocks nor log segments found. Creating new log.
I20260810 19:00:18.909381 12987 log.cc:824] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:00:18 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388418, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:00:18.911873 12987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749: No bootstrap required, opened a new log
I20260810 19:00:18.915676 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:18.898810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51826 (local address 127.27.147.190:35935)
0810 19:00:18.899621 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:18.899666 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:18.899786 (+ 120us) server_negotiation.cc:407] Connection header received
0810 19:00:18.901158 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:18.901184 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:18.901593 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:18.901965 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:00:18.902935 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:18.904602 (+ 1667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:18.905891 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:18.906732 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:18.906970 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:18.910261 (+ 3291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:00:18.910273 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:00:18.910293 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:00:18.910324 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:00:18.912835 (+ 2511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:18.913305 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:18.913310 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:18.913315 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:18.913426 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:18.913720 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:18.913724 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:18.913726 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:18.914103 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:00:18.914386 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:18.914613 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 19:00:18.914940 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":243,"threads_started":1}
I20260810 19:00:18.917273 12987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a006f6de9234a9db8015aa589b3b749" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35935 } }
I20260810 19:00:18.917544 12987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:00:18.917631 12987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a006f6de9234a9db8015aa589b3b749, State: Initialized, Role: FOLLOWER
I20260810 19:00:18.918114 12987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a006f6de9234a9db8015aa589b3b749" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35935 } }
I20260810 19:00:18.918295 12987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:00:18.918375 12987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:00:18.918512 12987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:00:18.920032 12987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a006f6de9234a9db8015aa589b3b749" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35935 } }
I20260810 19:00:18.920397 12987 leader_election.cc:302] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8a006f6de9234a9db8015aa589b3b749; no voters:
I20260810 19:00:18.920728 12987 leader_election.cc:288] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:00:18.920986 13000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:00:18.921361 13000 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [term 1 LEADER]: Becoming Leader. State: Replica: 8a006f6de9234a9db8015aa589b3b749, State: Running, Role: LEADER
I20260810 19:00:18.921721 13000 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a006f6de9234a9db8015aa589b3b749" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35935 } }
I20260810 19:00:18.922016 12987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:00:18.923030 13000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8a006f6de9234a9db8015aa589b3b749. Latest consensus state: current_term: 1 leader_uuid: "8a006f6de9234a9db8015aa589b3b749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a006f6de9234a9db8015aa589b3b749" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35935 } } }
I20260810 19:00:18.923179 13000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [sys.catalog]: This master's current role is: LEADER
I20260810 19:00:18.923895 13001 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8a006f6de9234a9db8015aa589b3b749" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8a006f6de9234a9db8015aa589b3b749" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35935 } } }
I20260810 19:00:18.924013 13001 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749 [sys.catalog]: This master's current role is: LEADER
I20260810 19:00:18.924849 13013 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:00:18.928726 13013 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:00:18.929230 12987 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:00:18.953469 13013 catalog_manager.cc:1347] Generated new cluster ID: 4ce2a3cc3b39436596e09e8225538367
I20260810 19:00:18.953548 13013 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:00:18.968426 13013 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:00:18.969725 13013 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:00:18.976519 13013 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749: Generated new TSK 0
I20260810 19:00:18.977396 13013 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 19:00:19.948165 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:54249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35935
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:00:20.128381 13059 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:00:20.128609 13059 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:00:20.128649 13059 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:00:20.132807 13059 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:00:20.132862 13059 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:00:20.132949 13059 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:00:20.138304 13059 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.147.148:54249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13059
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:00:20.139390 13059 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:00:20.140394 13059 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:00:20.152876 13072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:00:20.153043 13059 server_base.cc:1034] running on GCE node
W20260810 19:00:20.152848 13069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:00:20.153240 13076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:00:20.153782 13059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:00:20.154613 13059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:00:20.155838 13059 hybrid_clock.cc:648] HybridClock initialized: now 1786388420155801 us; error 52 us; skew 500 ppm
Aug 10 19:00:20 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:00:20.160290 13059 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:00:20.161309 13059 webserver.cc:460] Webserver started at http://127.27.147.129:41201/ using document root <none> and password file <none>
I20260810 19:00:20.161855 13059 fs_manager.cc:359] Metadata directory not provided
I20260810 19:00:20.161948 13059 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:00:20.162194 13059 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:00:20 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:00:20.180640 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "47837e2c16504dae9f6c8a931c2a8b42"
format_stamp: "Formatted at 2026-08-10 19:00:20 on dist-test-slave-znh6"
server_key: "debf87989f17da78b1f3b98d4ce5132a"
server_key_iv: "e9462cf8c7d2dbb9987813b9c1169c0e"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:20.182538 13059 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "47837e2c16504dae9f6c8a931c2a8b42"
format_stamp: "Formatted at 2026-08-10 19:00:20 on dist-test-slave-znh6"
server_key: "debf87989f17da78b1f3b98d4ce5132a"
server_key_iv: "e9462cf8c7d2dbb9987813b9c1169c0e"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:20.188707 13059 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20260810 19:00:20.208868 13088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:20.210554 13059 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 19:00:20.210697 13059 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "47837e2c16504dae9f6c8a931c2a8b42"
format_stamp: "Formatted at 2026-08-10 19:00:20 on dist-test-slave-znh6"
server_key: "debf87989f17da78b1f3b98d4ce5132a"
server_key_iv: "e9462cf8c7d2dbb9987813b9c1169c0e"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:20.210857 13059 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:00:20.233058 13059 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:00:20.237139 13059 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:00:20.237399 13059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:00:20.238186 13059 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:00:20.239398 13059 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:00:20.239507 13059 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:20.239629 13059 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:00:20.239703 13059 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:20.267283 13059 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:38701
I20260810 19:00:20.269025 13275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:38701 every 8 connection(s)
I20260810 19:00:20.269035 13059 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:00:20.277313 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 13059
I20260810 19:00:20.277458 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:00:20.277776 28238 external_mini_cluster.cc:1428] Setting key f495adb2b53df0529bd993a766cf3900
Aug 10 19:00:20 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:00:20.289906 13288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:20.272001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35935 (local address 127.27.147.129:41383)
0810 19:00:20.272755 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:20.272814 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:20.273894 (+ 1080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:20.274644 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:20.274656 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:20.275143 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:00:20.275761 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:20.275777 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.277009 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.277013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:20.278800 (+ 1787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:20.278840 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.280201 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.280206 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:20.280548 (+ 342us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:20.281401 (+ 853us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:00:20.281425 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:00:20.283504 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:00:20.286144 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:00:20.286151 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:00:20.286167 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:00:20.286561 (+ 394us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:00:20.286969 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:00:20.286973 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:00:20.286975 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:00:20.287155 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:00:20.287720 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:00:20.287727 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:00:20.288104 (+ 377us) client_negotiation.cc:769] Sending connection context
0810 19:00:20.288303 (+ 199us) client_negotiation.cc:240] Negotiation successful
0810 19:00:20.288632 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":258,"threads_started":1}
I20260810 19:00:20.291028 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:20.271528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:41383 (local address 127.27.147.190:35935)
0810 19:00:20.273945 (+ 2417us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:20.273950 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:20.273970 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:00:20.274210 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:20.274214 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:20.274272 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:20.274388 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:00:20.275933 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.276824 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.279065 (+ 2241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.279481 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.279604 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:20.283703 (+ 4099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:00:20.283707 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:00:20.283710 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:00:20.283736 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:00:20.286001 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:20.286686 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:20.286690 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:20.286691 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:20.286840 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:20.287275 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:20.287279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:20.287281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:20.287560 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:00:20.287661 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:20.290538 (+ 2877us) server_negotiation.cc:299] Negotiation successful
0810 19:00:20.290808 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2328,"thread_start_us":96,"threads_started":1}
I20260810 19:00:20.295077 13282 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35935
I20260810 19:00:20.295394 13282 heartbeater.cc:464] Registering TS with master...
I20260810 19:00:20.296182 13282 heartbeater.cc:511] Master 127.27.147.190:35935 requested a full tablet report, sending...
I20260810 19:00:20.298808 12932 ts_manager.cc:194] Registered new tserver with Master: 47837e2c16504dae9f6c8a931c2a8b42 (127.27.147.129:38701)
I20260810 19:00:20.300850 12932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:41383
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 19:00:20.336542 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:54249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35935
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:00:20 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 19:00:20.580415 13291 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:00:20.580801 13291 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:00:20.580897 13291 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:00:20.585685 13291 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:00:20.585777 13291 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:00:20.585889 13291 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:00:20.591388 13291 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.147.148:54249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13291
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:00:20.592505 13291 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:00:20.593547 13291 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:00:20.604048 13300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:00:20.604102 13304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:00:20.606163 13291 server_base.cc:1034] running on GCE node
W20260810 19:00:20.607182 13301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:00:20.607702 13291 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:00:20.608513 13291 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:00:20.609723 13291 hybrid_clock.cc:648] HybridClock initialized: now 1786388420609672 us; error 76 us; skew 500 ppm
Aug 10 19:00:20 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:00:20.614405 13291 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:00:20.615705 13291 webserver.cc:460] Webserver started at http://127.27.147.130:33619/ using document root <none> and password file <none>
I20260810 19:00:20.616583 13291 fs_manager.cc:359] Metadata directory not provided
I20260810 19:00:20.616688 13291 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:00:20.617118 13291 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:00:20 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:00:20.633761 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "af1676b6425a439dac3a9e33a8cc38d3"
format_stamp: "Formatted at 2026-08-10 19:00:20 on dist-test-slave-znh6"
server_key: "494f4f3c9ae27b823504e39b92424735"
server_key_iv: "2e0a3558d5fd00bbf8b277ce6c4b0a6b"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:20.634518 13291 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "af1676b6425a439dac3a9e33a8cc38d3"
format_stamp: "Formatted at 2026-08-10 19:00:20 on dist-test-slave-znh6"
server_key: "494f4f3c9ae27b823504e39b92424735"
server_key_iv: "2e0a3558d5fd00bbf8b277ce6c4b0a6b"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:20.639933 13291 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 19:00:20.658443 13315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:20.660115 13291 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:00:20.660248 13291 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "af1676b6425a439dac3a9e33a8cc38d3"
format_stamp: "Formatted at 2026-08-10 19:00:20 on dist-test-slave-znh6"
server_key: "494f4f3c9ae27b823504e39b92424735"
server_key_iv: "2e0a3558d5fd00bbf8b277ce6c4b0a6b"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:20.660368 13291 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:00:20.691803 13291 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:00:20.695391 13291 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:00:20.695645 13291 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:00:20.696421 13291 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:00:20.697494 13291 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:00:20.697593 13291 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:20.697687 13291 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:00:20.697750 13291 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:20.720566 13291 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:45917
I20260810 19:00:20.721786 13291 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:00:20.724205 13507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:45917 every 8 connection(s)
I20260810 19:00:20.731108 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 13291
I20260810 19:00:20.731236 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:00:20.731539 28238 external_mini_cluster.cc:1428] Setting key 63656516b0c851a81f2ec9b1b8686d1f
Aug 10 19:00:20 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:00:20.746122 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:20.726305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:53823 (local address 127.27.147.190:35935)
0810 19:00:20.726454 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:20.726459 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:20.728366 (+ 1907us) server_negotiation.cc:407] Connection header received
0810 19:00:20.729729 (+ 1363us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:20.729732 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:20.729792 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:20.729901 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:00:20.732668 (+ 2767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.733443 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.737663 (+ 4220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.738020 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.738135 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:20.741649 (+ 3514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:00:20.741653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:00:20.741656 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:00:20.741684 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:00:20.743671 (+ 1987us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:20.744267 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:20.744271 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:20.744273 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:20.744376 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:20.744899 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:20.744903 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:20.744905 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:20.745130 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:00:20.745239 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:20.745826 (+ 587us) server_negotiation.cc:299] Negotiation successful
0810 19:00:20.745959 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:00:20.747365 13511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:20.726809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35935 (local address 127.27.147.130:53823)
0810 19:00:20.728190 (+ 1381us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:20.728247 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:20.729463 (+ 1216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:20.730105 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:20.730115 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:20.730576 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:00:20.732491 (+ 1915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:20.732512 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.736064 (+ 3552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.736068 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:20.737541 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:20.737553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:20.738192 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:20.738197 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:20.738839 (+ 642us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:20.739554 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:00:20.739575 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:00:20.741466 (+ 1891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:00:20.743789 (+ 2323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:00:20.743795 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:00:20.743811 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:00:20.744147 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:00:20.744603 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:00:20.744606 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:00:20.744608 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:00:20.744789 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:00:20.745254 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:00:20.745261 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:00:20.745553 (+ 292us) client_negotiation.cc:769] Sending connection context
0810 19:00:20.745748 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 19:00:20.746052 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1111,"thread_start_us":209,"threads_started":1}
I20260810 19:00:20.749011 13509 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35935
I20260810 19:00:20.749255 13509 heartbeater.cc:464] Registering TS with master...
I20260810 19:00:20.749786 13509 heartbeater.cc:511] Master 127.27.147.190:35935 requested a full tablet report, sending...
I20260810 19:00:20.750976 12932 ts_manager.cc:194] Registered new tserver with Master: af1676b6425a439dac3a9e33a8cc38d3 (127.27.147.130:45917)
I20260810 19:00:20.751772 12932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:53823
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 19:00:20.786041 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:54249/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35935
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:46521
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:00:20.999434 13517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:00:20.999764 13517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:00:20.999825 13517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:00:21.006426 13517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:00:21.006605 13517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:00:21.006793 13517 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:00:21.015130 13517 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:46521
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.147.148:54249/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35935
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13517
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:00:21.016893 13517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:00:21.018256 13517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:00:21.032312 13526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:00:21.033259 13530 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:00:21.033501 13517 server_base.cc:1034] running on GCE node
W20260810 19:00:21.033958 13525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:00:21.034482 13517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:00:21.035233 13517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:00:21.036418 13517 hybrid_clock.cc:648] HybridClock initialized: now 1786388421036357 us; error 74 us; skew 500 ppm
Aug 10 19:00:21 dist-test-slave-znh6 krb5kdc[12166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:00:21.040633 13517 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:00:21.041724 13517 webserver.cc:460] Webserver started at http://127.27.147.131:36887/ using document root <none> and password file <none>
I20260810 19:00:21.042464 13517 fs_manager.cc:359] Metadata directory not provided
I20260810 19:00:21.042623 13517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:00:21.042929 13517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:00:21 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:00:21.059551 13517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "5f70c2c75dfe4d2eb44a727fa34122b0"
format_stamp: "Formatted at 2026-08-10 19:00:21 on dist-test-slave-znh6"
server_key: "7bab68ea9dbb5fbeaa4db8ede503bcc1"
server_key_iv: "fae5570710e1e0cb16aaebe61bce8070"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:21.061125 13517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "5f70c2c75dfe4d2eb44a727fa34122b0"
format_stamp: "Formatted at 2026-08-10 19:00:21 on dist-test-slave-znh6"
server_key: "7bab68ea9dbb5fbeaa4db8ede503bcc1"
server_key_iv: "fae5570710e1e0cb16aaebe61bce8070"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:21.073740 13517 fs_manager.cc:691] Time spent creating directory manager: real 0.012s user 0.000s sys 0.004s
I20260810 19:00:21.130095 13539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:21.132172 13517 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.001s sys 0.000s
I20260810 19:00:21.132328 13517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "5f70c2c75dfe4d2eb44a727fa34122b0"
format_stamp: "Formatted at 2026-08-10 19:00:21 on dist-test-slave-znh6"
server_key: "7bab68ea9dbb5fbeaa4db8ede503bcc1"
server_key_iv: "fae5570710e1e0cb16aaebe61bce8070"
server_key_version: "kuduclusterkey@0"
I20260810 19:00:21.132476 13517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Aug 10 19:00:21 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:00:21.160805 13517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:00:21.168210 13517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:00:21.168407 13517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:00:21.169013 13517 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:00:21.169966 13517 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:00:21.170013 13517 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:21.170061 13517 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:00:21.170082 13517 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:00:21.196267 13517 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:36975
I20260810 19:00:21.197582 13517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:00:21.201284 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 13517
I20260810 19:00:21.201413 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:00:21.201768 28238 external_mini_cluster.cc:1428] Setting key 518142c0b7917594806792c7cf2996eb
I20260810 19:00:21.207926 13730 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:36975 every 8 connection(s)
Aug 10 19:00:21 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388421, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:00:21.231730 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.212600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:56801 (local address 127.27.147.190:35935)
0810 19:00:21.212714 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.212730 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.212753 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:00:21.214171 (+ 1418us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.214175 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.214231 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.214316 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:00:21.217664 (+ 3348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.218514 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.220235 (+ 1721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.220589 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.220700 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.227393 (+ 6693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:00:21.227397 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:00:21.227400 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:00:21.227430 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:00:21.229279 (+ 1849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:21.229919 (+ 640us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:21.229922 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:21.229924 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:21.230013 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:21.230420 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:21.230423 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:21.230425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:21.230645 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:00:21.230726 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.231429 (+ 703us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.231566 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:00:21.233423 13735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.211380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35935 (local address 127.27.147.131:56801)
0810 19:00:21.212126 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:21.212190 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:21.213664 (+ 1474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:21.214527 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:21.214536 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:21.216709 (+ 2173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:00:21.217512 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.217534 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.218636 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.218640 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.220115 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.220128 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.220743 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.220748 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.220976 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.222316 (+ 1340us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:00:21.222341 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:00:21.227200 (+ 4859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:00:21.229415 (+ 2215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:00:21.229422 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:00:21.229438 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:00:21.229798 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:00:21.230125 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:00:21.230128 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:00:21.230130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:00:21.230309 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:00:21.230806 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:00:21.230831 (+ 25us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:00:21.231137 (+ 306us) client_negotiation.cc:769] Sending connection context
0810 19:00:21.231975 (+ 838us) client_negotiation.cc:240] Negotiation successful
0810 19:00:21.232315 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":245,"threads_started":1}
I20260810 19:00:21.234828 13733 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35935
I20260810 19:00:21.235091 13733 heartbeater.cc:464] Registering TS with master...
I20260810 19:00:21.235697 13733 heartbeater.cc:511] Master 127.27.147.190:35935 requested a full tablet report, sending...
I20260810 19:00:21.236896 12932 ts_manager.cc:194] Registered new tserver with Master: 5f70c2c75dfe4d2eb44a727fa34122b0 (127.27.147.131:36975)
I20260810 19:00:21.237778 12932 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:56801
I20260810 19:00:21.241577 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:00:21.274703 13287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.253560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51840 (local address 127.27.147.190:35935)
0810 19:00:21.253698 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.253703 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.253718 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:00:21.253756 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.253759 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.253802 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.253877 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:00:21.254872 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.255687 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.256938 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.257285 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.257356 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.270958 (+ 13602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:00:21.270962 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:00:21.270965 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:00:21.270995 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:00:21.272797 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:21.273338 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:21.273341 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:21.273343 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:21.273429 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:00:21.273724 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:00:21.273727 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:00:21.273728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:00:21.273939 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:00:21.274052 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.274429 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.274544 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:00:21.278475 12932 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51840:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:00:21.283864 12932 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:00:21.304641 13282 heartbeater.cc:503] Master 127.27.147.190:35935 was elected leader, sending a full tablet report...
I20260810 19:00:21.330480 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.313385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:38701 (local address 127.0.0.1:56836)
0810 19:00:21.323483 (+ 10098us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:21.323497 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:21.324271 (+ 774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:21.325079 (+ 808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:21.325085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:21.325109 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:00:21.325243 (+ 134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.325248 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.326830 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.326834 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.328265 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.328279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.329901 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.329905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.330018 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.330060 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 19:00:21.330185 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 19:00:21.330323 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10030,"thread_start_us":199,"threads_started":1}
I20260810 19:00:21.330981 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.313530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56836 (local address 127.27.147.129:38701)
0810 19:00:21.323294 (+ 9764us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.323303 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.324190 (+ 887us) server_negotiation.cc:407] Connection header received
0810 19:00:21.324447 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.324454 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.324682 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.324835 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:00:21.325376 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.326665 (+ 1289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.328420 (+ 1755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.329314 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.329444 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.329588 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.330648 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.330760 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9619,"thread_start_us":159,"threads_started":1}
I20260810 19:00:21.334486 13170 tablet_service.cc:1467] Processing CreateTablet for tablet e9c4ef4b309849d08cdc73f062cb2199 (DEFAULT_TABLE table=test-table [id=38ddc4311e464cd99e9698691422235c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:00:21.335860 13170 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c4ef4b309849d08cdc73f062cb2199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:00:21.336686 13754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.313236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:36975 (local address 127.0.0.1:53080)
0810 19:00:21.316254 (+ 3018us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:21.317112 (+ 858us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:21.317249 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:21.331984 (+ 14735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:21.331990 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:21.332011 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:00:21.332089 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.332095 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.333686 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.333689 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.334951 (+ 1262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.334962 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.336216 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.336220 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.336395 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.336422 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:00:21.336474 (+ 52us) client_negotiation.cc:240] Negotiation successful
0810 19:00:21.336541 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2815,"thread_start_us":121,"threads_started":1}
I20260810 19:00:21.337379 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.314127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53080 (local address 127.27.147.131:36975)
0810 19:00:21.330104 (+ 15977us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.330137 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.330165 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:00:21.330246 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.330251 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.330459 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.332279 (+ 1820us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:00:21.332424 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.333565 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.335223 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.336075 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.336903 (+ 828us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.337031 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.337080 (+ 49us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.337165 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15856,"thread_start_us":252,"threads_started":1}
I20260810 19:00:21.339687 13751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.323677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:45917 (local address 127.0.0.1:34642)
0810 19:00:21.330663 (+ 6986us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:21.330678 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:21.330742 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:21.331793 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:21.331799 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:21.331819 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:00:21.331897 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.331904 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.333393 (+ 1489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.333396 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.334585 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.334597 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.339293 (+ 4696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.339297 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.339388 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.339412 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:00:21.339491 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:00:21.339560 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6930,"thread_start_us":188,"threads_started":1}
I20260810 19:00:21.340176 13764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.323844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34642 (local address 127.27.147.130:45917)
0810 19:00:21.331193 (+ 7349us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.331201 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.331227 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:00:21.331314 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.331320 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.331501 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.331642 (+ 141us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:00:21.332040 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.333251 (+ 1211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.336663 (+ 3412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.338947 (+ 2284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.339084 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.339221 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.339888 (+ 667us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.339991 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7235,"thread_start_us":165,"threads_started":1}
I20260810 19:00:21.340655 13624 tablet_service.cc:1467] Processing CreateTablet for tablet e9c4ef4b309849d08cdc73f062cb2199 (DEFAULT_TABLE table=test-table [id=38ddc4311e464cd99e9698691422235c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:00:21.341732 13624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c4ef4b309849d08cdc73f062cb2199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:00:21.345785 13391 tablet_service.cc:1467] Processing CreateTablet for tablet e9c4ef4b309849d08cdc73f062cb2199 (DEFAULT_TABLE table=test-table [id=38ddc4311e464cd99e9698691422235c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:00:21.347028 13391 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9c4ef4b309849d08cdc73f062cb2199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:00:21.351534 13765 tablet_bootstrap.cc:492] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: Bootstrap starting.
I20260810 19:00:21.355150 13765 tablet_bootstrap.cc:654] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: Neither blocks nor log segments found. Creating new log.
I20260810 19:00:21.356722 13765 log.cc:824] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: Log is configured to *not* fsync() on all Append() calls
I20260810 19:00:21.363704 13768 tablet_bootstrap.cc:492] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Bootstrap starting.
I20260810 19:00:21.364966 13765 tablet_bootstrap.cc:492] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: No bootstrap required, opened a new log
I20260810 19:00:21.365190 13765 ts_tablet_manager.cc:1397] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.006s
I20260810 19:00:21.366925 13768 tablet_bootstrap.cc:654] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Neither blocks nor log segments found. Creating new log.
I20260810 19:00:21.367964 13768 log.cc:824] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Log is configured to *not* fsync() on all Append() calls
I20260810 19:00:21.370241 13768 tablet_bootstrap.cc:492] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: No bootstrap required, opened a new log
I20260810 19:00:21.370440 13768 ts_tablet_manager.cc:1397] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.002s
I20260810 19:00:21.375244 13767 tablet_bootstrap.cc:492] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Bootstrap starting.
I20260810 19:00:21.375443 13768 raft_consensus.cc:348] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.375732 13768 raft_consensus.cc:374] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:00:21.375788 13768 raft_consensus.cc:729] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f70c2c75dfe4d2eb44a727fa34122b0, State: Initialized, Role: FOLLOWER
I20260810 19:00:21.376245 13768 consensus_queue.cc:260] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.377205 13733 heartbeater.cc:503] Master 127.27.147.190:35935 was elected leader, sending a full tablet report...
I20260810 19:00:21.377302 13768 ts_tablet_manager.cc:1428] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20260810 19:00:21.378226 13767 tablet_bootstrap.cc:654] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Neither blocks nor log segments found. Creating new log.
I20260810 19:00:21.379473 13767 log.cc:824] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Log is configured to *not* fsync() on all Append() calls
I20260810 19:00:21.384680 13765 raft_consensus.cc:348] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.384966 13765 raft_consensus.cc:374] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:00:21.385025 13765 raft_consensus.cc:729] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47837e2c16504dae9f6c8a931c2a8b42, State: Initialized, Role: FOLLOWER
I20260810 19:00:21.385438 13767 tablet_bootstrap.cc:492] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: No bootstrap required, opened a new log
I20260810 19:00:21.385623 13767 ts_tablet_manager.cc:1397] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.000s
I20260810 19:00:21.385545 13765 consensus_queue.cc:260] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.387056 13765 ts_tablet_manager.cc:1428] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: Time spent starting tablet: real 0.022s user 0.010s sys 0.000s
I20260810 19:00:21.394968 13767 raft_consensus.cc:348] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.395269 13767 raft_consensus.cc:374] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:00:21.395355 13767 raft_consensus.cc:729] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af1676b6425a439dac3a9e33a8cc38d3, State: Initialized, Role: FOLLOWER
I20260810 19:00:21.395856 13767 consensus_queue.cc:260] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.396639 13509 heartbeater.cc:503] Master 127.27.147.190:35935 was elected leader, sending a full tablet report...
I20260810 19:00:21.399672 13767 ts_tablet_manager.cc:1428] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Time spent starting tablet: real 0.014s user 0.002s sys 0.007s
W20260810 19:00:21.463876 13734 tablet.cc:2369] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:00:21.477262 13510 tablet.cc:2369] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:00:21.521346 13283 tablet.cc:2369] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 19:00:21 dist-test-slave-znh6 krb5kdc[12166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388420, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:00:21.773926 13776 raft_consensus.cc:482] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:00:21.774202 13776 raft_consensus.cc:504] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.775672 13776 leader_election.cc:288] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f70c2c75dfe4d2eb44a727fa34122b0 (127.27.147.131:36975), 47837e2c16504dae9f6c8a931c2a8b42 (127.27.147.129:38701)
I20260810 19:00:21.781764 13758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.776268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:42441 (local address 127.27.147.131:36975)
0810 19:00:21.776616 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.776623 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.777250 (+ 627us) server_negotiation.cc:407] Connection header received
0810 19:00:21.777484 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.777487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.777548 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.777604 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:00:21.778043 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.778842 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.780445 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.781043 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.781152 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.781218 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.781555 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.781631 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":245}
I20260810 19:00:21.782395 13783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.775874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:36975 (local address 127.27.147.130:42441)
0810 19:00:21.777158 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:21.777175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:21.777335 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:21.777776 (+ 441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:21.777778 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:21.777792 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:00:21.777914 (+ 122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.777921 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.779006 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.779009 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.780303 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.780314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.781318 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.781322 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.781407 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.781429 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 19:00:21.782152 (+ 723us) client_negotiation.cc:240] Negotiation successful
0810 19:00:21.782237 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"mutex_wait_us":7,"thread_start_us":113,"threads_started":1}
I20260810 19:00:21.782506 13755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.777280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:54547 (local address 127.27.147.129:38701)
0810 19:00:21.777408 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.777414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.777715 (+ 301us) server_negotiation.cc:407] Connection header received
0810 19:00:21.777900 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.777903 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.777964 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.778018 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:00:21.778418 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.779159 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.781199 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.781763 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.781877 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.781931 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.782299 (+ 368us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.782369 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:00:21.783007 13781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.776839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:38701 (local address 127.27.147.130:54547)
0810 19:00:21.777621 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:21.777635 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:00:21.777780 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:00:21.778151 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:00:21.778154 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:00:21.778169 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:00:21.778289 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.778295 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.779847 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.779851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.781061 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:00:21.781073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.782068 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.782071 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:00:21.782161 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.782180 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 19:00:21.782678 (+ 498us) client_negotiation.cc:240] Negotiation successful
0810 19:00:21.782756 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":713,"thread_start_us":150,"threads_started":1}
I20260810 19:00:21.783304 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c4ef4b309849d08cdc73f062cb2199" candidate_uuid: "af1676b6425a439dac3a9e33a8cc38d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" is_pre_election: true
I20260810 19:00:21.783601 13662 raft_consensus.cc:2454] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af1676b6425a439dac3a9e33a8cc38d3 in term 0.
I20260810 19:00:21.783632 13180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c4ef4b309849d08cdc73f062cb2199" candidate_uuid: "af1676b6425a439dac3a9e33a8cc38d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47837e2c16504dae9f6c8a931c2a8b42" is_pre_election: true
I20260810 19:00:21.783946 13180 raft_consensus.cc:2454] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af1676b6425a439dac3a9e33a8cc38d3 in term 0.
I20260810 19:00:21.784021 13320 leader_election.cc:302] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f70c2c75dfe4d2eb44a727fa34122b0, af1676b6425a439dac3a9e33a8cc38d3; no voters:
I20260810 19:00:21.784253 13776 raft_consensus.cc:2781] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:00:21.784340 13776 raft_consensus.cc:482] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:00:21.784385 13776 raft_consensus.cc:3037] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:00:21.785599 13776 raft_consensus.cc:504] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.785984 13776 leader_election.cc:288] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [CANDIDATE]: Term 1 election: Requested vote from peers 5f70c2c75dfe4d2eb44a727fa34122b0 (127.27.147.131:36975), 47837e2c16504dae9f6c8a931c2a8b42 (127.27.147.129:38701)
I20260810 19:00:21.786291 13662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c4ef4b309849d08cdc73f062cb2199" candidate_uuid: "af1676b6425a439dac3a9e33a8cc38d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0"
I20260810 19:00:21.786404 13662 raft_consensus.cc:3037] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:00:21.786674 13180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9c4ef4b309849d08cdc73f062cb2199" candidate_uuid: "af1676b6425a439dac3a9e33a8cc38d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47837e2c16504dae9f6c8a931c2a8b42"
I20260810 19:00:21.786787 13180 raft_consensus.cc:3037] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:00:21.787950 13662 raft_consensus.cc:2454] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1676b6425a439dac3a9e33a8cc38d3 in term 1.
I20260810 19:00:21.788370 13320 leader_election.cc:302] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f70c2c75dfe4d2eb44a727fa34122b0, af1676b6425a439dac3a9e33a8cc38d3; no voters:
I20260810 19:00:21.788623 13180 raft_consensus.cc:2454] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af1676b6425a439dac3a9e33a8cc38d3 in term 1.
I20260810 19:00:21.789443 13776 raft_consensus.cc:2781] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:00:21.789657 13776 raft_consensus.cc:686] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 1 LEADER]: Becoming Leader. State: Replica: af1676b6425a439dac3a9e33a8cc38d3, State: Running, Role: LEADER
I20260810 19:00:21.789994 13776 consensus_queue.cc:237] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } }
I20260810 19:00:21.793689 12932 catalog_manager.cc:5495] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to af1676b6425a439dac3a9e33a8cc38d3 (127.27.147.130). New cstate: current_term: 1 leader_uuid: "af1676b6425a439dac3a9e33a8cc38d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af1676b6425a439dac3a9e33a8cc38d3" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 45917 } health_report { overall_health: HEALTHY } } }
I20260810 19:00:21.897010 13799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.891823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34652 (local address 127.27.147.130:45917)
0810 19:00:21.892048 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 19:00:21.892064 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:00:21.892423 (+ 359us) server_negotiation.cc:407] Connection header received
0810 19:00:21.892600 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:00:21.892603 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:00:21.892673 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:00:21.892798 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:00:21.893058 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.893897 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.895203 (+ 1306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:00:21.895602 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:00:21.895705 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:00:21.896043 (+ 338us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:00:21.896561 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:00:21.896674 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:00:21.896761 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 19:00:21.896816 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":58,"threads_started":1}
I20260810 19:00:21.903927 13662 raft_consensus.cc:1261] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 1 FOLLOWER]: Refusing update from remote peer af1676b6425a439dac3a9e33a8cc38d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:00:21.903973 13180 raft_consensus.cc:1261] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 1 FOLLOWER]: Refusing update from remote peer af1676b6425a439dac3a9e33a8cc38d3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:00:21.904704 13785 consensus_queue.cc:1035] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f70c2c75dfe4d2eb44a727fa34122b0" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 36975 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:00:21.904927 13776 consensus_queue.cc:1035] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47837e2c16504dae9f6c8a931c2a8b42" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:00:21.912046 13801 mvcc.cc:205] Tried to move back new op lower bound from 7317046976110571520 to 7317046975656108032. Current Snapshot: MvccSnapshot[applied={T|T < 7317046976110571520}]
I20260810 19:00:21.916862 13809 mvcc.cc:205] Tried to move back new op lower bound from 7317046976110571520 to 7317046975656108032. Current Snapshot: MvccSnapshot[applied={T|T < 7317046976110571520}]
I20260810 19:00:21.918680 13810 mvcc.cc:205] Tried to move back new op lower bound from 7317046976110571520 to 7317046975656108032. Current Snapshot: MvccSnapshot[applied={T|T < 7317046976110571520}]
I20260810 19:00:21.923727 12931 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:00:21.923897 12931 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51840:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:00:21.927054 12931 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8a006f6de9234a9db8015aa589b3b749: Sending DeleteTablet for 3 replicas of tablet e9c4ef4b309849d08cdc73f062cb2199
I20260810 19:00:21.927796 13624 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c4ef4b309849d08cdc73f062cb2199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:00:21 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:53080
I20260810 19:00:21.928212 13170 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c4ef4b309849d08cdc73f062cb2199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:00:21 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:56836
I20260810 19:00:21.928759 13824 tablet_replica.cc:330] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42: stopping tablet replica
I20260810 19:00:21.928763 13391 tablet_service.cc:1514] Processing DeleteTablet for tablet e9c4ef4b309849d08cdc73f062cb2199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:00:21 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:34642
I20260810 19:00:21.929180 13823 tablet_replica.cc:330] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: stopping tablet replica
I20260810 19:00:21.929237 13824 raft_consensus.cc:2229] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:00:21.929636 13824 raft_consensus.cc:2258] T e9c4ef4b309849d08cdc73f062cb2199 P 47837e2c16504dae9f6c8a931c2a8b42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:00:21.929947 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 13059
I20260810 19:00:21.939707 13825 tablet_replica.cc:330] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: stopping tablet replica
I20260810 19:00:21.940119 13825 raft_consensus.cc:2229] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:00:21.940659 13825 raft_consensus.cc:2258] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:00:21.942227 13825 ts_tablet_manager.cc:1905] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:00:21.929476 13823 raft_consensus.cc:2229] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:00:21.943147 13823 raft_consensus.cc:2258] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:00:21.945168 13823 ts_tablet_manager.cc:1905] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:00:21.946225 13825 ts_tablet_manager.cc:1918] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:00:21.946472 13825 log.cc:1196] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/e9c4ef4b309849d08cdc73f062cb2199
I20260810 19:00:21.946946 13825 ts_tablet_manager.cc:1939] T e9c4ef4b309849d08cdc73f062cb2199 P af1676b6425a439dac3a9e33a8cc38d3: Deleting consensus metadata
I20260810 19:00:21.948096 13823 ts_tablet_manager.cc:1918] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:00:21.948199 13823 log.cc:1196] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/e9c4ef4b309849d08cdc73f062cb2199
I20260810 19:00:21.948254 12914 catalog_manager.cc:4841] TS af1676b6425a439dac3a9e33a8cc38d3 (127.27.147.130:45917): tablet e9c4ef4b309849d08cdc73f062cb2199 (table test-table [id=38ddc4311e464cd99e9698691422235c]) successfully deleted
I20260810 19:00:21.948522 13823 ts_tablet_manager.cc:1939] T e9c4ef4b309849d08cdc73f062cb2199 P 5f70c2c75dfe4d2eb44a727fa34122b0: Deleting consensus metadata
I20260810 19:00:21.949558 12916 catalog_manager.cc:4841] TS 5f70c2c75dfe4d2eb44a727fa34122b0 (127.27.147.131:36975): tablet e9c4ef4b309849d08cdc73f062cb2199 (table test-table [id=38ddc4311e464cd99e9698691422235c]) successfully deleted
W20260810 19:00:21.952556 12918 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:38701 (error 108)
I20260810 19:00:21.953711 13827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:00:21.953296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:38701 (local address 127.0.0.1:56842)
0810 19:00:21.953537 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0810 19:00:21.953611 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:38701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":64,"threads_started":1}
I20260810 19:00:21.953796 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 13291
W20260810 19:00:21.965561 12918 catalog_manager.cc:4584] TS 47837e2c16504dae9f6c8a931c2a8b42 (127.27.147.129:38701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e9c4ef4b309849d08cdc73f062cb2199: Network error: Client connection negotiation failed: client connection to 127.27.147.129:38701: connect: Connection refused (error 111)
I20260810 19:00:21.972508 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 13517
I20260810 19:00:21.992093 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 12886
I20260810 19:00:22.009375 28238 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 19:00:22.354534 28238 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 19:00:22.354665 28238 mini_ranger.cc:66] Stopping Ranger...
I20260810 19:00:22.710067 28238 mini_ranger.cc:68] Stopped Ranger
2026-08-10 19:00:22.710 UTC [12214] LOG: received smart shutdown request
2026-08-10 19:00:22.714 UTC [12214] LOG: background worker "logical replication launcher" (PID 12225) exited with exit code 1
2026-08-10 19:00:22.715 UTC [12218] LOG: shutting down
2026-08-10 19:00:22.901 UTC [12214] LOG: database system is shut down
2026-08-10T19:00:22Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49847 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:00:23 dist-test-slave-znh6 krb5kdc[13844](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:00:23 dist-test-slave-znh6 krb5kdc[13844](info): set up 2 sockets
Aug 10 19:00:23 dist-test-slave-znh6 krb5kdc[13844](info): commencing operation
krb5kdc: starting...
W20260810 19:00:24.292022 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.252s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:00:24 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388424, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:00:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:00:24Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/rangeradmin_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/rangeradmin_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/rangerlookup_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/rangerlookup_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
I20260810 19:00:24.469789 28238 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskfzwUfv/build/debug/bin/postgres/pg_ctl -D /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/postgres -l logfile start
2026-08-10 19:00:26.899 UTC [13898] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 19:00:26.899 UTC [13898] LOG: listening on IPv4 address "127.27.147.148", port 39613
2026-08-10 19:00:26.903 UTC [13898] LOG: listening on Unix socket "/tmp/.s.PGSQL.39613"
2026-08-10 19:00:26.924 UTC [13901] LOG: database system was shut down at 2026-08-10 19:00:25 UTC
2026-08-10 19:00:26.928 UTC [13898] LOG: database system is ready to accept connections
I20260810 19:00:28.081223 28238 mini_postgres.cc:96] Postgres bound to 39613
2026-08-10 19:00:28.089 UTC [13913] FATAL: database "slave" does not exist
127.27.147.148:39613 - accepting connections
I20260810 19:00:28.089558 28238 mini_ranger.cc:161] Starting Ranger...
I20260810 19:00:28.105237 28238 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 19:00:28.570681 28238 mini_ranger.cc:87] Created ranger Postgres database
I20260810 19:00:28.570755 28238 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-admin
2026-08-10 19:00:28,865 [I] DB FLAVOR :POSTGRES
2026-08-10 19:00:28,866 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:00:28,866 [I] Checking connection..
2026-08-10 19:00:28,866 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 19:00:29,107 [I] Checking connection passed.
2026-08-10 19:00:29,107 [I] --------- Verifying version history table ---------
2026-08-10 19:00:29,107 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:00:29,358 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:00:29,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:00:29,611 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 19:00:29,611 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 19:00:29,611 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 19:00:29.844 UTC [14028] WARNING: there is no transaction in progress
2026-08-10 19:00:29,853 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 19:00:29,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 19:00:30,123 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 19:00:30,124 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:00:30,124 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 19:00:30,373 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 19:00:30,643 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-znh6.c.gcp-upstream.internal','N') ;"
2026-08-10 19:00:30,895 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:00:31,158 [I] Table x_portal_user does not exist in database ranger
2026-08-10 19:00:31,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:00:31,390 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 19:00:31,391 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:00:31,647 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:00:31,898 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 19:00:31,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 19:00:32.868 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:32.883 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:32.899 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:32.913 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:32.928 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:32.945 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:32.997 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.003 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.016 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.025 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.035 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.048 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.057 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.063 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.070 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33.077 UTC [14196] WARNING: there is no transaction in progress
2026-08-10 19:00:33,720 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 19:00:33,721 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 19:00:33,966 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 19:00:33,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 19:00:34,216 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 19:00:34,217 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 19:00:34,479 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 19:00:34,480 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 19:00:34,743 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 19:00:34,975 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-znh6.c.gcp-upstream.internal';"
2026-08-10 19:00:35,228 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 19:00:35,228 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 19:00:35,493 [I] DB_PATCHES have already been applied
I20260810 19:00:35.498939 28238 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-admin:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskfzwUfv/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 19:00:35.498999 28238 mini_ranger.cc:193] Using host: 127.27.147.148
I20260810 19:00:35.503482 28238 mini_ranger.cc:227] Ranger admin URL: http://127.27.147.148:35477
Aug 10, 2026 7:00:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:00:35 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:00:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 7:00:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 7:00:35 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 7:00:36 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 7:00:36 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 19:00:36.875352 28238 mini_ranger.cc:160] Time spent starting Ranger: real 8.786s user 0.004s sys 0.008s
Aug 10, 2026 7:00:56 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 7:00:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 7:00:56 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 7:00:56 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 7:00:57 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 19:00:58.347527 28238 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.27.147.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.27.147.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/rangerkms_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.27.147.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/rangerkms_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.27.147.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.27.147.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.27.147.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/HTTP_127.27.147.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 19:00:58 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 19:00:58.457727 28238 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 19:00:58.468335 28238 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 19:00:59.344442 28238 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 19:00:59.344543 28238 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms
I20260810 19:00:59.344569 28238 mini_ranger_kms.cc:227] Using postgres at 127.27.147.148:39613
2026-08-10 19:00:59,537 [I] DB FLAVOR :POSTGRES
2026-08-10 19:00:59,538 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 19:00:59,538 [I] Checking connection
2026-08-10 19:00:59,538 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 19:00:59,805 [I] connection success
2026-08-10 19:00:59,805 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 19:00:59,805 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 19:00:59,806 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 19:01:00,060 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 19:01:00,060 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 19:01:00,061 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 19:01:00,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.27.147.148:39613/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 19:01:00,321 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 19:01:00.788738 28238 mini_ranger_kms.cc:310] Created kms service
I20260810 19:01:00.993088 28238 mini_ranger_kms.cc:326] Created kudu user
I20260810 19:01:01.039304 28238 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 19:01:01 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388458, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:01.422883 28238 mini_ranger_kms.cc:384] Added ranger policy
I20260810 19:01:01.423039 28238 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms:/tmp/dist-test-taskfzwUfv/build/debug/bin/postgresql.jar:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskfzwUfv/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 19:01:01.423060 28238 mini_ranger_kms.cc:240] Using host: 127.27.147.148
I20260810 19:01:01.428326 28238 mini_ranger_kms.cc:276] Ranger KMS PID: 14519
I20260810 19:01:01.428426 28238 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.27.147.148:41955
Aug 10, 2026 7:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 7:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 7:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 7:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 7:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 7:01:02 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 19:01:03 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786388463, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.147.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 19:01:03 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388463, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.147.148@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:04.328153 28238 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.870s user 0.012s sys 0.166s
I20260810 19:01:04.328281 28238 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 19:01:04.328351 28238 mini_ranger_kms.cc:401] 127.27.147.148:41955/kms/v1/keys
Aug 10 19:01:04 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388463, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.27.147.148@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
Aug 10 19:01:06 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:01:06.604405 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38637
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38637
--ranger_config_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:06.717756 14608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:06.717990 14608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:06.718029 14608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:06.722113 14608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:06.722164 14608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:06.722184 14608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:06.722200 14608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:06.722215 14608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:06.727540 14608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
--ranger_kms_url=127.27.147.148:41955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38637
--ranger_config_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14608
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:06.728464 14608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:06.729477 14608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:06.739002 14621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:06.739058 14625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:06.739039 14622 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:06.739266 14608 server_base.cc:1034] running on GCE node
I20260810 19:01:06.739910 14608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:06.740960 14608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:06.742161 14608 hybrid_clock.cc:648] HybridClock initialized: now 1786388466742144 us; error 33 us; skew 500 ppm
Aug 10 19:01:06 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:06.750012 14608 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:06.751065 14608 webserver.cc:460] Webserver started at http://127.27.147.190:44147/ using document root <none> and password file <none>
I20260810 19:01:06.751605 14608 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:06.751662 14608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:06.751884 14608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:01:06 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388466, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:06.799716 14608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "5623b0878f524952849306d8fe29c8db"
format_stamp: "Formatted at 2026-08-10 19:01:06 on dist-test-slave-znh6"
server_key: "732a322daee63d49c9ab80b653b02831"
server_key_iv: "894b52431e72727b82fafeb1a969c87f"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:06.800457 14608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "5623b0878f524952849306d8fe29c8db"
format_stamp: "Formatted at 2026-08-10 19:01:06 on dist-test-slave-znh6"
server_key: "732a322daee63d49c9ab80b653b02831"
server_key_iv: "894b52431e72727b82fafeb1a969c87f"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:06.804823 14608 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:01:06.818486 14635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:06.819837 14608 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 19:01:06.819974 14608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "5623b0878f524952849306d8fe29c8db"
format_stamp: "Formatted at 2026-08-10 19:01:06 on dist-test-slave-znh6"
server_key: "732a322daee63d49c9ab80b653b02831"
server_key_iv: "894b52431e72727b82fafeb1a969c87f"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:06.820096 14608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:06.841639 14608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:06.844607 14608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:06.844841 14608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:06.853367 14608 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38637
I20260810 19:01:06.853389 14724 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38637 every 8 connection(s)
I20260810 19:01:06.854327 14608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:06.857187 14726 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:06.861582 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 14608
I20260810 19:01:06.861713 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:01:06.862023 28238 external_mini_cluster.cc:1428] Setting key 5900180784cc1763e381aa9c799a021b
I20260810 19:01:06.863497 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Bootstrap starting.
I20260810 19:01:06.865877 14726 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:06.866943 14726 log.cc:824] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:06.868965 14726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: No bootstrap required, opened a new log
Aug 10 19:01:06 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388466, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:06.873421 14726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:06.873732 14726 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:06.873828 14726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5623b0878f524952849306d8fe29c8db, State: Initialized, Role: FOLLOWER
I20260810 19:01:06.874483 14726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:06.874665 14726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:06.874770 14726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:06.874963 14726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:06.876137 14726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:06.876521 14726 leader_election.cc:302] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5623b0878f524952849306d8fe29c8db; no voters:
I20260810 19:01:06.876924 14726 leader_election.cc:288] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:01:06.877024 14734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:06.877280 14734 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 LEADER]: Becoming Leader. State: Replica: 5623b0878f524952849306d8fe29c8db, State: Running, Role: LEADER
I20260810 19:01:06.878257 14726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:06.878383 14734 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:06.879175 14730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:06.863609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49654 (local address 127.27.147.190:38637)
0810 19:01:06.864108 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:06.864132 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:06.864175 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:01:06.865045 (+ 870us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:06.865072 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:06.865512 (+ 440us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:06.865881 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:06.866860 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:06.868387 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:06.869581 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:06.870297 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:06.870458 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:06.873552 (+ 3094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:06.873558 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:06.873573 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:06.873604 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:06.875768 (+ 2164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:06.876353 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:06.876369 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:06.876375 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:06.876505 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:06.876834 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:06.876840 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:06.876845 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:06.877450 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:06.877647 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:06.877904 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 19:01:06.878175 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":147,"threads_started":1}
I20260810 19:01:06.880610 14726 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:01:06.881647 14735 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5623b0878f524952849306d8fe29c8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } } }
I20260810 19:01:06.881810 14735 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:06.881657 14734 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5623b0878f524952849306d8fe29c8db. Latest consensus state: current_term: 1 leader_uuid: "5623b0878f524952849306d8fe29c8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } } }
I20260810 19:01:06.882345 14734 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:06.885143 14756 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:06.889731 14756 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:06.896669 14756 catalog_manager.cc:1347] Generated new cluster ID: b2892011d0a0429ca3b60be6cf72f257
I20260810 19:01:06.896724 14756 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:06.915750 14756 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:01:06.917150 14756 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:06.925360 14756 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Generated new TSK 0
I20260810 19:01:06.926237 14756 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 19:01:07.621527 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38637
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:07.767825 14799 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:07.768083 14799 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:07.768126 14799 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:07.772334 14799 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:07.772394 14799 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:07.772475 14799 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:01:07.779991 14799 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.27.147.148:41955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14799
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:07.781461 14799 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:07.782765 14799 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:07.794157 14811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:07.794337 14813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:07.794696 14799 server_base.cc:1034] running on GCE node
W20260810 19:01:07.794163 14810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:07.795176 14799 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:07.795922 14799 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:07.797111 14799 hybrid_clock.cc:648] HybridClock initialized: now 1786388467797047 us; error 78 us; skew 500 ppm
Aug 10 19:01:07 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388467, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:07.800761 14799 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:01:07.801821 14799 webserver.cc:460] Webserver started at http://127.27.147.129:42799/ using document root <none> and password file <none>
I20260810 19:01:07.802523 14799 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:07.802610 14799 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:07.802934 14799 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:01:07 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388467, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:07.834995 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "5a17c6cf91b943708621f99cb14f6881"
format_stamp: "Formatted at 2026-08-10 19:01:07 on dist-test-slave-znh6"
server_key: "e2686dd60070729bf79a9ca5c96f9237"
server_key_iv: "dcbad5c91998a197107ffbacf6ec88f8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:07.836305 14799 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "5a17c6cf91b943708621f99cb14f6881"
format_stamp: "Formatted at 2026-08-10 19:01:07 on dist-test-slave-znh6"
server_key: "e2686dd60070729bf79a9ca5c96f9237"
server_key_iv: "dcbad5c91998a197107ffbacf6ec88f8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:07.843487 14799 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.000s
I20260810 19:01:07.867978 14827 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:07.869761 14799 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:01:07.869982 14799 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "5a17c6cf91b943708621f99cb14f6881"
format_stamp: "Formatted at 2026-08-10 19:01:07 on dist-test-slave-znh6"
server_key: "e2686dd60070729bf79a9ca5c96f9237"
server_key_iv: "dcbad5c91998a197107ffbacf6ec88f8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:07.870206 14799 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:07.895994 14799 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:07.899904 14799 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:07.900168 14799 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:07.901010 14799 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:07.902125 14799 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:07.902240 14799 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:07.902375 14799 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:07.902472 14799 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:07.916008 14799 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:41679
I20260810 19:01:07.917204 14799 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:01:07.918808 15015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:41679 every 8 connection(s)
I20260810 19:01:07.924386 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 14799
I20260810 19:01:07.924527 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:01:07.924877 28238 external_mini_cluster.cc:1428] Setting key c84247fc2a5a58b1ddb0b68fe345b81d
Aug 10 19:01:07 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786388467, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Aug 10 19:01:07 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388467, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:07.944643 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:07.922138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:51155 (local address 127.27.147.190:38637)
0810 19:01:07.923533 (+ 1395us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:07.923542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:07.923561 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:01:07.929252 (+ 5691us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:07.929256 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:07.929313 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:07.929397 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:07.931018 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:07.931896 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:07.933665 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:07.934040 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:07.934114 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:07.937824 (+ 3710us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:07.937828 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:07.937831 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:07.937858 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:07.939849 (+ 1991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:07.942769 (+ 2920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:07.942773 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:07.942775 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:07.942897 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:07.943322 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:07.943326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:07.943328 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:07.943551 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:07.943625 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:07.944310 (+ 685us) server_negotiation.cc:299] Negotiation successful
0810 19:01:07.944447 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1314,"thread_start_us":99,"threads_started":1}
I20260810 19:01:07.946444 15022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:07.921027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38637 (local address 127.27.147.129:51155)
0810 19:01:07.923070 (+ 2043us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:07.923130 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:07.926274 (+ 3144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:07.929796 (+ 3522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:07.929805 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:07.930149 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:07.930848 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:07.930868 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:07.932111 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:07.932114 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:07.933522 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:07.933534 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:07.934323 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:07.934327 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:07.934518 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:07.935238 (+ 720us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:07.935260 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:07.937607 (+ 2347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:07.942286 (+ 4679us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:07.942291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:07.942308 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:07.942636 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:07.943010 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:07.943013 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:07.943015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:07.943185 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:07.943707 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:07.943713 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:07.944026 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 19:01:07.945093 (+ 1067us) client_negotiation.cc:240] Negotiation successful
0810 19:01:07.945427 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1817,"thread_start_us":138,"threads_started":1}
I20260810 19:01:07.952224 15016 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38637
I20260810 19:01:07.952481 15016 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:07.953081 15016 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:07.954921 14663 ts_manager.cc:194] Registered new tserver with Master: 5a17c6cf91b943708621f99cb14f6881 (127.27.147.129:41679)
I20260810 19:01:07.956575 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:51155
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 19:01:07.997192 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38637
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:08.165412 15027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:08.165934 15027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:08.166085 15027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:08.173058 15027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:08.173143 15027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:08.173249 15027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:01:08.182155 15027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.27.147.148:41955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15027
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:08.183903 15027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:08.185441 15027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:08.198741 15041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:08.198750 15035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:08.198750 15033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:08.200065 15027 server_base.cc:1034] running on GCE node
I20260810 19:01:08.200559 15027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:08.201304 15027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:08.202533 15027 hybrid_clock.cc:648] HybridClock initialized: now 1786388468202501 us; error 45 us; skew 500 ppm
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:08.207301 15027 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:01:08.209858 15027 webserver.cc:460] Webserver started at http://127.27.147.130:35599/ using document root <none> and password file <none>
I20260810 19:01:08.210675 15027 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:08.210798 15027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:08.211124 15027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:08.227284 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "84e43b1f07714866a693dcbe6d5517c6"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "8631dc779c95f5403deb201568b32940"
server_key_iv: "ab4957bea21bd83b39eb172f2c7ec1b8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:08.228071 15027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "84e43b1f07714866a693dcbe6d5517c6"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "8631dc779c95f5403deb201568b32940"
server_key_iv: "ab4957bea21bd83b39eb172f2c7ec1b8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:08.235630 15027 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260810 19:01:08.256392 15052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:08.258169 15027 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.002s sys 0.000s
I20260810 19:01:08.258315 15027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "84e43b1f07714866a693dcbe6d5517c6"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "8631dc779c95f5403deb201568b32940"
server_key_iv: "ab4957bea21bd83b39eb172f2c7ec1b8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:08.258461 15027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:08.278136 15027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:08.287926 15027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:08.288153 15027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:08.289067 15027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:08.290139 15027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:08.290207 15027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:08.290262 15027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:08.290292 15027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:08.310879 15027 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:42245
I20260810 19:01:08.311049 15226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:42245 every 8 connection(s)
I20260810 19:01:08.312287 15027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:01:08.320123 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 15027
I20260810 19:01:08.320255 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:01:08.320632 28238 external_mini_cluster.cc:1428] Setting key ac1bf65db6bfdf6a17c10a3f4299036a
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
I20260810 19:01:08.342087 15233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:08.315265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38637 (local address 127.27.147.130:57477)
0810 19:01:08.316863 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:08.316922 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:08.318296 (+ 1374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:08.319164 (+ 868us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:08.319173 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:08.319694 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:08.320451 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:08.320473 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.321671 (+ 1198us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.321675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:08.323148 (+ 1473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:08.323160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.324041 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.324045 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:08.324295 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:08.325027 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:08.325048 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:08.326894 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:08.337868 (+ 10974us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:08.337876 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:08.337899 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:08.338329 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:08.338780 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:08.338784 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:08.338788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:08.339097 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:08.339665 (+ 568us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:08.339676 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:08.340211 (+ 535us) client_negotiation.cc:769] Sending connection context
0810 19:01:08.340529 (+ 318us) client_negotiation.cc:240] Negotiation successful
0810 19:01:08.340912 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1362,"thread_start_us":184,"threads_started":1}
I20260810 19:01:08.343267 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:08.314755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:57477 (local address 127.27.147.190:38637)
0810 19:01:08.314957 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:08.314962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:08.317036 (+ 2074us) server_negotiation.cc:407] Connection header received
0810 19:01:08.318571 (+ 1535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:08.318575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:08.318637 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:08.318700 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:08.320642 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.321497 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.323325 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.323709 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.323796 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:08.327094 (+ 3298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:08.327098 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:08.327101 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:08.327130 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:08.328988 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:08.338535 (+ 9547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:08.338539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:08.338542 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:08.338652 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:08.339275 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:08.339279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:08.339281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:08.339546 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:08.339662 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:08.342899 (+ 3237us) server_negotiation.cc:299] Negotiation successful
0810 19:01:08.343051 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 19:01:08.344381 15229 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38637
I20260810 19:01:08.344660 15229 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:08.345397 15229 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:08.346684 14663 ts_manager.cc:194] Registered new tserver with Master: 84e43b1f07714866a693dcbe6d5517c6 (127.27.147.130:42245)
I20260810 19:01:08.347505 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:57477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 19:01:08.363081 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38637
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388467, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
W20260810 19:01:08.569825 15236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:08.570219 15236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:08.570302 15236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:08.576984 15236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:08.577064 15236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:08.577158 15236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:01:08.586084 15236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.27.147.148:41955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15236
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:08.587604 15236 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:08.589033 15236 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:08.603174 15246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:08.603324 15244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:08.603188 15242 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:08.603653 15236 server_base.cc:1034] running on GCE node
I20260810 19:01:08.607362 15236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:08.608151 15236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:08.611471 15236 hybrid_clock.cc:648] HybridClock initialized: now 1786388468611433 us; error 46 us; skew 500 ppm
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:08.615352 15236 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:01:08.616425 15236 webserver.cc:460] Webserver started at http://127.27.147.131:44013/ using document root <none> and password file <none>
I20260810 19:01:08.617123 15236 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:08.617202 15236 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:08.617426 15236 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:08.634047 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "025e8938767147a0b399e29242bd757a"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "85b6f238a2b75f8469287ddef91cdf2f"
server_key_iv: "9f765cb760c615526c134b312b53b322"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:08.634938 15236 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "025e8938767147a0b399e29242bd757a"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "85b6f238a2b75f8469287ddef91cdf2f"
server_key_iv: "9f765cb760c615526c134b312b53b322"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:08.640877 15236 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 19:01:08.659226 15255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:08.661060 15236 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.000s
I20260810 19:01:08.661207 15236 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "025e8938767147a0b399e29242bd757a"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "85b6f238a2b75f8469287ddef91cdf2f"
server_key_iv: "9f765cb760c615526c134b312b53b322"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:08.661341 15236 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:08.677410 15236 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:08.681362 15236 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:08.681617 15236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:08.682394 15236 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:08.683558 15236 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:08.683666 15236 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:08.683784 15236 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:08.683853 15236 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:08.696321 15236 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:40909
I20260810 19:01:08.696770 15436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:40909 every 8 connection(s)
I20260810 19:01:08.697579 15236 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:01:08.705029 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 15236
I20260810 19:01:08.705173 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:01:08.705524 28238 external_mini_cluster.cc:1428] Setting key af9cd812889d75ae430257f4d336f505
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:08.716240 15024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:08.701495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:50047 (local address 127.27.147.190:38637)
0810 19:01:08.701635 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:08.701641 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:08.701909 (+ 268us) server_negotiation.cc:407] Connection header received
0810 19:01:08.703153 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:08.703157 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:08.703216 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:08.703313 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:08.705052 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.705900 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.707691 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.708059 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.708139 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:08.711668 (+ 3529us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:08.711672 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:08.711675 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:08.711703 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:08.713564 (+ 1861us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:08.714322 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:08.714326 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:08.714329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:08.714430 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:08.714918 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:08.714923 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:08.714924 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:08.715169 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:08.715285 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:08.715939 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 19:01:08.716069 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260810 19:01:08.717101 15440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:08.700230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38637 (local address 127.27.147.131:50047)
0810 19:01:08.701702 (+ 1472us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:08.701764 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:08.702883 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:08.703760 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:08.703770 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:08.704156 (+ 386us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:08.704862 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:08.704884 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.706079 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.706082 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:08.707541 (+ 1459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:08.707553 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:08.708354 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:08.708358 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:08.708550 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:08.709289 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:08.709311 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:08.711278 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:08.713720 (+ 2442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:08.713728 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:08.713760 (+ 32us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:08.714161 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:08.714557 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:08.714561 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:08.714562 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:08.714744 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:08.715300 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:08.715307 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:08.715605 (+ 298us) client_negotiation.cc:769] Sending connection context
0810 19:01:08.715825 (+ 220us) client_negotiation.cc:240] Negotiation successful
0810 19:01:08.716134 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1217,"thread_start_us":176,"threads_started":1}
I20260810 19:01:08.718328 15437 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38637
I20260810 19:01:08.718565 15437 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:08.719167 15437 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:08.720409 14663 ts_manager.cc:194] Registered new tserver with Master: 025e8938767147a0b399e29242bd757a (127.27.147.131:40909)
I20260810 19:01:08.721158 14663 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:50047
I20260810 19:01:08.721524 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:01:08.722240 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 14799
I20260810 19:01:08.763697 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 15027
I20260810 19:01:08.779464 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 15236
I20260810 19:01:08.795703 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 14608
I20260810 19:01:08.809070 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38637
--webserver_interface=127.27.147.190
--webserver_port=44147
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38637
--ranger_config_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:08.921931 15442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:08.922163 15442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:08.922202 15442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:08.926319 15442 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:08.926373 15442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:08.926393 15442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:08.926409 15442 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:08.926462 15442 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:08.931696 15442 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38637
--ranger_config_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=44147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:08.932629 15442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:08.933584 15442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:08.942004 15458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:08.942210 15454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:08.942253 15442 server_base.cc:1034] running on GCE node
W20260810 19:01:08.942408 15461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:08.942982 15442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:08.944034 15442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:08.945235 15442 hybrid_clock.cc:648] HybridClock initialized: now 1786388468945208 us; error 37 us; skew 500 ppm
Aug 10 19:01:08 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:08.948031 15442 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:08.949164 15442 webserver.cc:460] Webserver started at http://127.27.147.190:44147/ using document root <none> and password file <none>
I20260810 19:01:08.949705 15442 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:08.949806 15442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388468, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:09.032001 15442 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 19:01:09.034917 15475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.036489 15442 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:01:09.036619 15442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "5623b0878f524952849306d8fe29c8db"
format_stamp: "Formatted at 2026-08-10 19:01:06 on dist-test-slave-znh6"
server_key: "732a322daee63d49c9ab80b653b02831"
server_key_iv: "894b52431e72727b82fafeb1a969c87f"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:09.037303 15442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:09.057106 15442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:09.060012 15442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:09.060164 15442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:09.067242 15442 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38637
I20260810 19:01:09.067251 15559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38637 every 8 connection(s)
I20260810 19:01:09.068351 15442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:09.072670 15560 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:01:09.074281 15560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Bootstrap starting.
I20260810 19:01:09.077211 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 15442
I20260810 19:01:09.077719 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:41679
--local_ip_for_outbound_sockets=127.27.147.129
--tserver_master_addrs=127.27.147.190:38637
--webserver_port=42799
--webserver_interface=127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:09.091946 15560 log.cc:824] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:09.098542 15560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:01:09.099009 15560 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Bootstrap complete.
I20260810 19:01:09.103363 15560 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:09.103741 15560 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5623b0878f524952849306d8fe29c8db, State: Initialized, Role: FOLLOWER
I20260810 19:01:09.104283 15560 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:09.104408 15560 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:09.104497 15560 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:09.104617 15560 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:01:09.106441 15560 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:09.106770 15560 leader_election.cc:302] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5623b0878f524952849306d8fe29c8db; no voters:
I20260810 19:01:09.107153 15560 leader_election.cc:288] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:01:09.107234 15569 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:01:09.107432 15569 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [term 2 LEADER]: Becoming Leader. State: Replica: 5623b0878f524952849306d8fe29c8db, State: Running, Role: LEADER
I20260810 19:01:09.107784 15569 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } }
I20260810 19:01:09.108402 15560 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:09.109342 15570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "5623b0878f524952849306d8fe29c8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } } }
I20260810 19:01:09.109483 15570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:09.109349 15571 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5623b0878f524952849306d8fe29c8db. Latest consensus state: current_term: 2 leader_uuid: "5623b0878f524952849306d8fe29c8db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5623b0878f524952849306d8fe29c8db" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38637 } } }
I20260810 19:01:09.109797 15571 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:09.110630 15581 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:09.110852 15560 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 19:01:09.116942 15581 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:09.117865 15581 catalog_manager.cc:1259] Loaded cluster ID: b2892011d0a0429ca3b60be6cf72f257
I20260810 19:01:09.117941 15581 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:09.120604 15581 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:09.121397 15581 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Loaded TSK: 0
I20260810 19:01:09.122669 15581 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:01:09.219228 15563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:09.219496 15563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:09.219590 15563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:09.225864 15563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:09.225975 15563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:09.226074 15563 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:01:09.232028 15563 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:41679
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=42799
--tserver_master_addrs=127.27.147.190:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15563
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:09.233037 15563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:09.234037 15563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:09.244762 15622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:09.244815 15623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:09.244877 15625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:09.245913 15563 server_base.cc:1034] running on GCE node
I20260810 19:01:09.246356 15563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:09.247112 15563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:09.250427 15563 hybrid_clock.cc:648] HybridClock initialized: now 1786388469250387 us; error 52 us; skew 500 ppm
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:09.254092 15563 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:01:09.255246 15563 webserver.cc:460] Webserver started at http://127.27.147.129:42799/ using document root <none> and password file <none>
I20260810 19:01:09.255962 15563 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:09.256143 15563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:09.291446 15563 fs_manager.cc:709] Time spent opening directory manager: real 0.009s user 0.003s sys 0.000s
I20260810 19:01:09.294476 15636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.295904 15563 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:01:09.296065 15563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "5a17c6cf91b943708621f99cb14f6881"
format_stamp: "Formatted at 2026-08-10 19:01:07 on dist-test-slave-znh6"
server_key: "e2686dd60070729bf79a9ca5c96f9237"
server_key_iv: "dcbad5c91998a197107ffbacf6ec88f8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:09.296623 15563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:09.324265 15563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:09.327814 15563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:09.328009 15563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:09.328678 15563 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:09.329723 15563 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:09.329792 15563 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.329855 15563 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:09.329882 15563 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.343530 15563 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:41679
I20260810 19:01:09.343675 15835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:41679 every 8 connection(s)
I20260810 19:01:09.344888 15563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:01:09.350564 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 15563
I20260810 19:01:09.351110 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:42245
--local_ip_for_outbound_sockets=127.27.147.130
--tserver_master_addrs=127.27.147.190:38637
--webserver_port=35599
--webserver_interface=127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:09.366030 15839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:09.347549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38637 (local address 127.27.147.129:36281)
0810 19:01:09.348351 (+ 802us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:09.348405 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:09.349456 (+ 1051us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:09.351412 (+ 1956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:09.351422 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:09.351788 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:09.352442 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:09.352464 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.354386 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.354390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:09.355836 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:09.355848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.357132 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.357136 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:09.357337 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:09.358028 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:09.358045 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:09.359811 (+ 1766us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:09.362329 (+ 2518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:09.362335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:09.362349 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:09.362709 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:09.363165 (+ 456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:09.363168 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:09.363170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:09.363368 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:09.364134 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:09.364140 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:09.364451 (+ 311us) client_negotiation.cc:769] Sending connection context
0810 19:01:09.364645 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 19:01:09.364973 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":145,"threads_started":1}
I20260810 19:01:09.368492 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:09.347405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:36281 (local address 127.27.147.190:38637)
0810 19:01:09.348052 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:09.348074 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:09.349787 (+ 1713us) server_negotiation.cc:407] Connection header received
0810 19:01:09.350080 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:09.350093 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:09.350511 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:09.350897 (+ 386us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:09.352643 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.354221 (+ 1578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.355989 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.356742 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.356913 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:09.360028 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:09.360036 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:09.360051 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:09.360083 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:09.362184 (+ 2101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.362888 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.362894 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.362899 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.363042 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.363503 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.363507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.363509 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.363987 (+ 478us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:09.367103 (+ 3116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:09.367263 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 19:01:09.367568 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":126,"threads_started":1}
I20260810 19:01:09.371654 15836 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38637
I20260810 19:01:09.371896 15836 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.372475 15836 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
W20260810 19:01:09.374809 15836 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38637 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:01:09.374974 15836 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.375136 15836 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:09.376057 15836 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.376219 15836 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
W20260810 19:01:09.377043 15836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:01:09.513707 15841 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:09.514074 15841 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:09.514161 15841 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:09.520375 15841 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:09.520483 15841 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:09.520581 15841 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:01:09.528630 15841 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:42245
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=35599
--tserver_master_addrs=127.27.147.190:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15841
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:09.530210 15841 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:09.531692 15841 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:09.544333 15859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:09.544744 15850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:09.547524 15852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:09.548228 15841 server_base.cc:1034] running on GCE node
I20260810 19:01:09.548774 15841 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:09.549650 15841 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:09.550899 15841 hybrid_clock.cc:648] HybridClock initialized: now 1786388469550861 us; error 51 us; skew 500 ppm
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:09.554095 15841 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:01:09.555146 15841 webserver.cc:460] Webserver started at http://127.27.147.130:35599/ using document root <none> and password file <none>
I20260810 19:01:09.555739 15841 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:09.555905 15841 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:09.578301 15841 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.002s sys 0.001s
I20260810 19:01:09.581712 15871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.583374 15841 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 19:01:09.583500 15841 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "84e43b1f07714866a693dcbe6d5517c6"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "8631dc779c95f5403deb201568b32940"
server_key_iv: "ab4957bea21bd83b39eb172f2c7ec1b8"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:09.584208 15841 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:09.605247 15841 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:09.608860 15841 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:09.609105 15841 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:09.609867 15841 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:09.611222 15841 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:09.611292 15841 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.611351 15841 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:09.611384 15841 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.623215 15841 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:42245
I20260810 19:01:09.623296 16055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:42245 every 8 connection(s)
I20260810 19:01:09.624634 15841 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:01:09.630925 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 15841
I20260810 19:01:09.631453 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.27.147.148:41955/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:40909
--local_ip_for_outbound_sockets=127.27.147.131
--tserver_master_addrs=127.27.147.190:38637
--webserver_port=44013
--webserver_interface=127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:09.642709 16061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:09.627281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38637 (local address 127.27.147.130:47057)
0810 19:01:09.627901 (+ 620us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:09.627963 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:09.629059 (+ 1096us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:09.629706 (+ 647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:09.629714 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:09.630060 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:09.630735 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:09.630755 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.632089 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.632092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:09.633523 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:09.633535 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.634301 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.634306 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:09.634530 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:09.635252 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:09.635273 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:09.637271 (+ 1998us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:09.639431 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:09.639437 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:09.639453 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:09.639790 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:09.640094 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:09.640097 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:09.640099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:09.640281 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:09.640744 (+ 463us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:09.640751 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:09.641080 (+ 329us) client_negotiation.cc:769] Sending connection context
0810 19:01:09.641298 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 19:01:09.641648 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":154,"threads_started":1}
I20260810 19:01:09.643375 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:09.626719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:47057 (local address 127.27.147.190:38637)
0810 19:01:09.626895 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:09.626900 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:09.628069 (+ 1169us) server_negotiation.cc:407] Connection header received
0810 19:01:09.629328 (+ 1259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:09.629332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:09.629398 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:09.629512 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:09.630980 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.631920 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.633684 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.634054 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.634129 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:09.637449 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:09.637453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:09.637456 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:09.637486 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:09.639290 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.639910 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.639914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.639916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.639998 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.640391 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.640394 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.640396 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.640642 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:09.640735 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:09.643051 (+ 2316us) server_negotiation.cc:299] Negotiation successful
0810 19:01:09.643199 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:01:09.645017 16056 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38637
I20260810 19:01:09.645278 16056 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.645931 16056 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
W20260810 19:01:09.647864 16056 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38637 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:01:09.648003 16056 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.648204 16056 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:09.649199 16056 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.649398 16056 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
W20260810 19:01:09.650259 16056 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:01:09.772405 16063 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:09.772722 16063 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:09.772832 16063 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:09.777096 16063 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:09.777189 16063 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:09.777279 16063 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:01:09.784493 16063 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.27.147.148:41955/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:40909
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=44013
--tserver_master_addrs=127.27.147.190:38637
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16063
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:09.785590 16063 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:09.786623 16063 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:09.796926 16071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:09.796926 16076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:09.797465 16074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:09.797657 16063 server_base.cc:1034] running on GCE node
I20260810 19:01:09.798111 16063 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:09.798900 16063 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:09.802198 16063 hybrid_clock.cc:648] HybridClock initialized: now 1786388469802156 us; error 48 us; skew 500 ppm
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:09.805688 16063 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:01:09.806718 16063 webserver.cc:460] Webserver started at http://127.27.147.131:44013/ using document root <none> and password file <none>
I20260810 19:01:09.807327 16063 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:09.807444 16063 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:09.825757 16063 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:01:09.830274 16087 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.831936 16063 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.000s
I20260810 19:01:09.832110 16063 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "025e8938767147a0b399e29242bd757a"
format_stamp: "Formatted at 2026-08-10 19:01:08 on dist-test-slave-znh6"
server_key: "85b6f238a2b75f8469287ddef91cdf2f"
server_key_iv: "9f765cb760c615526c134b312b53b322"
server_key_version: "kuduclusterkey@0"
I20260810 19:01:09.832908 16063 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:09.847443 16063 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:09.852350 16063 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:09.852568 16063 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:09.853302 16063 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:09.854466 16063 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:09.854544 16063 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.854604 16063 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:09.854638 16063 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:09.866879 16063 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:40909
I20260810 19:01:09.868247 16063 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:01:09.869091 16277 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:40909 every 8 connection(s)
I20260810 19:01:09.870951 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 16063
Aug 10 19:01:09 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388469, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:09.882472 16283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:09.872444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49670 (local address 127.27.147.190:38637)
0810 19:01:09.872725 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:09.872731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:09.872749 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:01:09.872800 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:09.872803 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:09.872852 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:09.874289 (+ 1437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:09.874385 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.875196 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.876514 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.876835 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.878635 (+ 1800us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:09.878754 (+ 119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:09.878759 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:09.878762 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:09.878779 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:09.880739 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.881167 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.881172 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.881176 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.881265 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.881565 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.881569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.881571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.881783 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:09.881985 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:09.882140 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 19:01:09.882263 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":202,"thread_start_us":107,"threads_started":1}
I20260810 19:01:09.886688 15838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:09.871809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:56787 (local address 127.27.147.190:38637)
0810 19:01:09.872004 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:09.872010 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:09.872793 (+ 783us) server_negotiation.cc:407] Connection header received
0810 19:01:09.874027 (+ 1234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:09.874031 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:09.874078 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:09.874129 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:09.875703 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.876452 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.878161 (+ 1709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.878512 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.878597 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:09.882540 (+ 3943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:09.882545 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:09.882550 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:09.882581 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:09.884268 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.885131 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.885135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.885136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.885220 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:09.885663 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:09.885666 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:09.885668 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:09.885834 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:09.885938 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:09.886416 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 19:01:09.886523 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77,"spinlock_wait_cycles":47360}
I20260810 19:01:09.887573 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:09.870710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38637 (local address 127.27.147.131:56787)
0810 19:01:09.872623 (+ 1913us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:09.872677 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:09.873798 (+ 1121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:09.874529 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:09.874538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:09.874900 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:09.875543 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:09.875563 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.876595 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.876597 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:09.878001 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:09.878016 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:09.878861 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:09.878867 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:09.879087 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:09.879640 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:09.879663 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:09.881727 (+ 2064us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:09.884498 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:09.884503 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:09.884516 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:09.885023 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:09.885411 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:09.885414 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:09.885416 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:09.885555 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:09.885939 (+ 384us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:09.885946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:09.886181 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 19:01:09.886358 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 19:01:09.886645 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1669,"thread_start_us":142,"threads_started":1}
I20260810 19:01:09.888582 16278 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38637
I20260810 19:01:09.888818 16278 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.889297 16278 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
W20260810 19:01:09.890688 16278 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38637 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:01:09.890849 16278 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.891028 16278 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:09.891765 16278 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:09.891956 16278 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
W20260810 19:01:09.892694 16278 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 19:01:10 dist-test-slave-znh6 krb5kdc[13844](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786388470, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:10.377280 15836 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:10.377585 15836 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:10.378899 15499 ts_manager.cc:194] Registered new tserver with Master: 5a17c6cf91b943708621f99cb14f6881 (127.27.147.129:41679)
I20260810 19:01:10.380672 15499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:36281
Aug 10 19:01:10 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388470, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:10.650496 16056 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:10.650786 16056 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:10.651636 15499 ts_manager.cc:194] Registered new tserver with Master: 84e43b1f07714866a693dcbe6d5517c6 (127.27.147.130:42245)
I20260810 19:01:10.652402 15499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:47057
I20260810 19:01:10.892918 16278 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:10.893229 16278 heartbeater.cc:511] Master 127.27.147.190:38637 requested a full tablet report, sending...
I20260810 19:01:10.894091 15499 ts_manager.cc:194] Registered new tserver with Master: 025e8938767147a0b399e29242bd757a (127.27.147.131:40909)
I20260810 19:01:10.895010 15499 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:56787
I20260810 19:01:10.900142 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:01:10.914034 16311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:10.901722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.27.147.190:38637)
0810 19:01:10.901984 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:10.901989 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:10.903066 (+ 1077us) server_negotiation.cc:407] Connection header received
0810 19:01:10.903118 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:10.903123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:10.903178 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:10.903302 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:10.904762 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.905697 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.907176 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.907636 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.907748 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:10.910143 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:10.910150 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:10.910154 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:10.910187 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:10.912237 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:10.912727 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:10.912731 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:10.912732 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:10.912831 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:10.913122 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:10.913126 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:10.913128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:10.913312 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:10.913454 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:10.913696 (+ 242us) server_negotiation.cc:299] Negotiation successful
0810 19:01:10.913833 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":103,"threads_started":1}
I20260810 19:01:10.918150 15499 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:01:10.922297 15499 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
Aug 10 19:01:10 dist-test-slave-znh6 krb5kdc[13844](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388470, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for HTTP/127.27.147.148@KRBTEST.COM
I20260810 19:01:10.945627 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:10.933376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:42245 (local address 127.0.0.1:43724)
0810 19:01:10.939126 (+ 5750us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:10.939180 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:10.939363 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:10.940432 (+ 1069us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:10.940438 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:10.940465 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:10.940594 (+ 129us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:10.940602 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.942199 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.942203 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:10.943672 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:10.943684 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.945067 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.945071 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:10.945190 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:10.945232 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 19:01:10.945347 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 19:01:10.945464 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5562,"thread_start_us":103,"threads_started":1}
I20260810 19:01:10.947789 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:10.934181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43724 (local address 127.27.147.130:42245)
0810 19:01:10.935239 (+ 1058us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:10.935245 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:10.939782 (+ 4537us) server_negotiation.cc:407] Connection header received
0810 19:01:10.939874 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:10.939880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:10.940088 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:10.940239 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:10.940732 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.942063 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.943843 (+ 1780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.944727 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.944863 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:10.944998 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:10.947538 (+ 2540us) server_negotiation.cc:299] Negotiation successful
0810 19:01:10.947652 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":959,"thread_start_us":92,"threads_started":1}
I20260810 19:01:10.951004 15941 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd93e93540d4c28b2ca5cb342f5baa0 (DEFAULT_TABLE table=test-table [id=886e8a1f64474ae4b493121964031dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:10.952159 15941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd93e93540d4c28b2ca5cb342f5baa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:10.953166 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:10.934146 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:40909 (local address 127.0.0.1:53950)
0810 19:01:10.945891 (+ 11745us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:10.945905 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:10.945986 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:10.946703 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:10.946706 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:10.946723 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:10.946810 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:10.946835 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.950075 (+ 3240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.950079 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:10.950951 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:10.950963 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.952778 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.952782 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:10.952875 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:10.952901 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:01:10.952966 (+ 65us) client_negotiation.cc:240] Negotiation successful
0810 19:01:10.953032 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11688,"thread_start_us":119,"threads_started":1}
I20260810 19:01:10.958050 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:10.934636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53950 (local address 127.27.147.131:40909)
0810 19:01:10.939654 (+ 5018us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:10.939663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:10.946087 (+ 6424us) server_negotiation.cc:407] Connection header received
0810 19:01:10.946174 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:10.946180 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:10.946386 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:10.946531 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:10.948705 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.949928 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.951097 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.952462 (+ 1365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.952583 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:10.952701 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:10.957016 (+ 4315us) server_negotiation.cc:299] Negotiation successful
0810 19:01:10.957902 (+ 886us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4910,"thread_start_us":116,"threads_started":1}
I20260810 19:01:10.961019 16139 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd93e93540d4c28b2ca5cb342f5baa0 (DEFAULT_TABLE table=test-table [id=886e8a1f64474ae4b493121964031dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:10.962119 16139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd93e93540d4c28b2ca5cb342f5baa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:10.962332 16332 tablet_bootstrap.cc:492] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Bootstrap starting.
I20260810 19:01:10.964998 16332 tablet_bootstrap.cc:654] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:10.965984 16332 log.cc:824] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:10.968784 16332 tablet_bootstrap.cc:492] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: No bootstrap required, opened a new log
I20260810 19:01:10.968981 16332 ts_tablet_manager.cc:1397] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20260810 19:01:10.971679 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:10.934804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55660 (local address 127.27.147.129:41679)
0810 19:01:10.935142 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:10.935149 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:10.953498 (+ 18349us) server_negotiation.cc:407] Connection header received
0810 19:01:10.953717 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:10.953722 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:10.953928 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:10.954041 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:10.954452 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.955612 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.969799 (+ 14187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.970702 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.970872 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:10.971026 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:10.971416 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 19:01:10.971534 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":125,"threads_started":1}
I20260810 19:01:10.972734 16332 raft_consensus.cc:348] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:10.972143 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:10.934240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:41679 (local address 127.0.0.1:55660)
0810 19:01:10.953403 (+ 19163us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:10.953415 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:10.953573 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:10.954224 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:10.954227 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:10.954245 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:10.954314 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:10.954320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.955788 (+ 1468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.955791 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:10.956847 (+ 1056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:10.956859 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:10.971112 (+ 14253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:10.971116 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:10.971219 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:10.971256 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 19:01:10.971898 (+ 642us) client_negotiation.cc:240] Negotiation successful
0810 19:01:10.971995 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19109,"thread_start_us":99,"threads_started":1}
I20260810 19:01:10.972976 16332 raft_consensus.cc:374] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:10.973021 16332 raft_consensus.cc:729] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84e43b1f07714866a693dcbe6d5517c6, State: Initialized, Role: FOLLOWER
I20260810 19:01:10.973937 16332 consensus_queue.cc:260] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:10.975124 16332 ts_tablet_manager.cc:1428] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:01:10.975522 16056 heartbeater.cc:503] Master 127.27.147.190:38637 was elected leader, sending a full tablet report...
I20260810 19:01:10.976051 15724 tablet_service.cc:1467] Processing CreateTablet for tablet 6cd93e93540d4c28b2ca5cb342f5baa0 (DEFAULT_TABLE table=test-table [id=886e8a1f64474ae4b493121964031dff]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:10.977125 15724 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6cd93e93540d4c28b2ca5cb342f5baa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:10.990209 16340 tablet_bootstrap.cc:492] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Bootstrap starting.
I20260810 19:01:10.992890 16340 tablet_bootstrap.cc:654] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:10.993824 16340 log.cc:824] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:10.994654 16341 tablet_bootstrap.cc:492] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881: Bootstrap starting.
I20260810 19:01:10.997893 16341 tablet_bootstrap.cc:654] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:10.998778 16341 log.cc:824] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:11.001649 16340 tablet_bootstrap.cc:492] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: No bootstrap required, opened a new log
I20260810 19:01:11.001874 16340 ts_tablet_manager.cc:1397] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.000s
I20260810 19:01:11.005978 16340 raft_consensus.cc:348] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.006242 16340 raft_consensus.cc:374] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:11.006302 16340 raft_consensus.cc:729] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 025e8938767147a0b399e29242bd757a, State: Initialized, Role: FOLLOWER
I20260810 19:01:11.006775 16340 consensus_queue.cc:260] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.008102 16340 ts_tablet_manager.cc:1428] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:01:11.008536 16278 heartbeater.cc:503] Master 127.27.147.190:38637 was elected leader, sending a full tablet report...
I20260810 19:01:11.009117 16341 tablet_bootstrap.cc:492] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881: No bootstrap required, opened a new log
I20260810 19:01:11.009312 16341 ts_tablet_manager.cc:1397] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20260810 19:01:11.013159 16341 raft_consensus.cc:348] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.013402 16341 raft_consensus.cc:374] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:11.013492 16341 raft_consensus.cc:729] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a17c6cf91b943708621f99cb14f6881, State: Initialized, Role: FOLLOWER
I20260810 19:01:11.013948 16341 consensus_queue.cc:260] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.014590 15836 heartbeater.cc:503] Master 127.27.147.190:38637 was elected leader, sending a full tablet report...
I20260810 19:01:11.014921 16341 ts_tablet_manager.cc:1428] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881: Time spent starting tablet: real 0.006s user 0.003s sys 0.004s
I20260810 19:01:11.022727 16350 raft_consensus.cc:482] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:11.022971 16350 raft_consensus.cc:504] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.026664 16350 leader_election.cc:288] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84e43b1f07714866a693dcbe6d5517c6 (127.27.147.130:42245), 025e8938767147a0b399e29242bd757a (127.27.147.131:40909)
I20260810 19:01:11.031646 16348 raft_consensus.cc:482] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:11.032024 16348 raft_consensus.cc:504] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.035089 16348 leader_election.cc:288] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84e43b1f07714866a693dcbe6d5517c6 (127.27.147.130:42245), 5a17c6cf91b943708621f99cb14f6881 (127.27.147.129:41679)
I20260810 19:01:11.039772 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.033551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:42245 (local address 127.27.147.131:48663)
0810 19:01:11.033973 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:11.033987 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:11.034075 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:11.034697 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:11.034700 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:11.034721 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:11.034901 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.034909 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.036097 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.036101 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.037400 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.037411 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.038482 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.038486 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.038593 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.038620 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:01:11.038671 (+ 51us) client_negotiation.cc:240] Negotiation successful
0810 19:01:11.039604 (+ 933us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":784,"thread_start_us":101,"threads_started":1}
I20260810 19:01:11.040066 16357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.033903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:48663 (local address 127.27.147.130:42245)
0810 19:01:11.034366 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:11.034375 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:11.034389 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:01:11.034432 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:11.034434 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:11.034478 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:11.035017 (+ 539us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:11.035146 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.035962 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.037571 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.038345 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.038870 (+ 525us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.039434 (+ 564us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:11.039474 (+ 40us) server_negotiation.cc:299] Negotiation successful
0810 19:01:11.039537 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":310,"thread_start_us":107,"threads_started":1}
I20260810 19:01:11.040745 15978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd93e93540d4c28b2ca5cb342f5baa0" candidate_uuid: "025e8938767147a0b399e29242bd757a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e43b1f07714866a693dcbe6d5517c6" is_pre_election: true
I20260810 19:01:11.041064 15978 raft_consensus.cc:2454] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025e8938767147a0b399e29242bd757a in term 0.
I20260810 19:01:11.041596 16092 leader_election.cc:302] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e8938767147a0b399e29242bd757a, 84e43b1f07714866a693dcbe6d5517c6; no voters:
I20260810 19:01:11.041929 16348 raft_consensus.cc:2781] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:01:11.042030 16348 raft_consensus.cc:482] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:01:11.042090 16348 raft_consensus.cc:3037] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:11.043845 16348 raft_consensus.cc:504] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.047176 15978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd93e93540d4c28b2ca5cb342f5baa0" candidate_uuid: "025e8938767147a0b399e29242bd757a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e43b1f07714866a693dcbe6d5517c6"
I20260810 19:01:11.047295 15978 raft_consensus.cc:3037] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:11.048570 15978 raft_consensus.cc:2454] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025e8938767147a0b399e29242bd757a in term 1.
I20260810 19:01:11.048921 16092 leader_election.cc:302] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 025e8938767147a0b399e29242bd757a, 84e43b1f07714866a693dcbe6d5517c6; no voters:
I20260810 19:01:11.049271 16360 raft_consensus.cc:2781] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:11.049554 16360 raft_consensus.cc:686] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 1 LEADER]: Becoming Leader. State: Replica: 025e8938767147a0b399e29242bd757a, State: Running, Role: LEADER
I20260810 19:01:11.049351 16348 leader_election.cc:288] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [CANDIDATE]: Term 1 election: Requested vote from peers 84e43b1f07714866a693dcbe6d5517c6 (127.27.147.130:42245), 5a17c6cf91b943708621f99cb14f6881 (127.27.147.129:41679)
I20260810 19:01:11.050148 16360 consensus_queue.cc:237] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } }
I20260810 19:01:11.050884 16328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.034953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:60433 (local address 127.27.147.129:41679)
0810 19:01:11.035055 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:11.035059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:11.040271 (+ 5212us) server_negotiation.cc:407] Connection header received
0810 19:01:11.044046 (+ 3775us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:11.044050 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:11.044096 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:11.044143 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:11.044462 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.045215 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.049469 (+ 4254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.050080 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.050168 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.050215 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:11.050652 (+ 437us) server_negotiation.cc:299] Negotiation successful
0810 19:01:11.050731 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:01:11.051249 16356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.034581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:41679 (local address 127.27.147.131:60433)
0810 19:01:11.040205 (+ 5624us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:11.040217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:11.043449 (+ 3232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:11.044260 (+ 811us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:11.044263 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:11.044278 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:11.044347 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.044352 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.045355 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.045358 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.046492 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.046503 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.050422 (+ 3919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.050425 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.050517 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.050539 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 19:01:11.051054 (+ 515us) client_negotiation.cc:240] Negotiation successful
0810 19:01:11.051128 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5571,"thread_start_us":100,"threads_started":1}
I20260810 19:01:11.051911 15733 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd93e93540d4c28b2ca5cb342f5baa0" candidate_uuid: "025e8938767147a0b399e29242bd757a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a17c6cf91b943708621f99cb14f6881" is_pre_election: true
I20260810 19:01:11.052109 15733 raft_consensus.cc:2454] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 025e8938767147a0b399e29242bd757a in term 0.
I20260810 19:01:11.052377 15751 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd93e93540d4c28b2ca5cb342f5baa0" candidate_uuid: "025e8938767147a0b399e29242bd757a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5a17c6cf91b943708621f99cb14f6881"
I20260810 19:01:11.052479 15751 raft_consensus.cc:3037] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:11.053658 15751 raft_consensus.cc:2454] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 025e8938767147a0b399e29242bd757a in term 1.
I20260810 19:01:11.055430 15498 catalog_manager.cc:5495] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a reported cstate change: term changed from 0 to 1, leader changed from <none> to 025e8938767147a0b399e29242bd757a (127.27.147.131). New cstate: current_term: 1 leader_uuid: "025e8938767147a0b399e29242bd757a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "025e8938767147a0b399e29242bd757a" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40909 } health_report { overall_health: HEALTHY } } }
I20260810 19:01:11.059213 16321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.025605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:37041 (local address 127.27.147.130:42245)
0810 19:01:11.025755 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:11.025760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:11.027186 (+ 1426us) server_negotiation.cc:407] Connection header received
0810 19:01:11.027639 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:11.027642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:11.027692 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:11.027739 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:11.028810 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.029591 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.037419 (+ 7828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.038085 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.038185 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.038248 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:11.059004 (+ 20756us) server_negotiation.cc:299] Negotiation successful
0810 19:01:11.059101 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:01:11.059732 16354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.025130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:42245 (local address 127.27.147.129:37041)
0810 19:01:11.027099 (+ 1969us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:11.027115 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:11.027503 (+ 388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:11.027993 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:11.027996 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:11.028009 (+ 13us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:11.028157 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.028162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.029738 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.029741 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.037274 (+ 7533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.037287 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.038392 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.038395 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.038503 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.038531 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:01:11.059511 (+ 20980us) client_negotiation.cc:240] Negotiation successful
0810 19:01:11.059596 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1897,"thread_start_us":103,"threads_started":1}
I20260810 19:01:11.060163 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.026528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:60433 (local address 127.27.147.131:40909)
0810 19:01:11.026632 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:11.026636 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:11.027425 (+ 789us) server_negotiation.cc:407] Connection header received
0810 19:01:11.028293 (+ 868us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:11.028296 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:11.028344 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:11.028390 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:11.030465 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.031260 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.038663 (+ 7403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.039305 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.039396 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.039437 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:11.059976 (+ 20539us) server_negotiation.cc:299] Negotiation successful
0810 19:01:11.060060 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:01:11.060995 15978 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd93e93540d4c28b2ca5cb342f5baa0" candidate_uuid: "5a17c6cf91b943708621f99cb14f6881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84e43b1f07714866a693dcbe6d5517c6" is_pre_election: true
I20260810 19:01:11.061172 15978 raft_consensus.cc:2379] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 5a17c6cf91b943708621f99cb14f6881 in current term 1: Already voted for candidate 025e8938767147a0b399e29242bd757a in this term.
I20260810 19:01:11.060616 16355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.026170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:40909 (local address 127.27.147.129:60433)
0810 19:01:11.027346 (+ 1176us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:11.027358 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:11.027872 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:11.028538 (+ 666us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:11.028541 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:11.028558 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:11.028668 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.028673 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.031402 (+ 2729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.031405 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.036475 (+ 5070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:11.036487 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.058679 (+ 22192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.058682 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:11.058807 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.058864 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 19:01:11.060406 (+ 1542us) client_negotiation.cc:240] Negotiation successful
0810 19:01:11.060486 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1118,"thread_start_us":111,"threads_started":1}
I20260810 19:01:11.062595 16207 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6cd93e93540d4c28b2ca5cb342f5baa0" candidate_uuid: "5a17c6cf91b943708621f99cb14f6881" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "025e8938767147a0b399e29242bd757a" is_pre_election: true
I20260810 19:01:11.063280 15638 leader_election.cc:302] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 5a17c6cf91b943708621f99cb14f6881; no voters: 025e8938767147a0b399e29242bd757a, 84e43b1f07714866a693dcbe6d5517c6
I20260810 19:01:11.063591 16350 raft_consensus.cc:2726] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 19:01:11.098140 15837 tablet.cc:2369] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:01:11.100152 16325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.095208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53952 (local address 127.27.147.131:40909)
0810 19:01:11.095331 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:11.095336 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:11.095349 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:01:11.095430 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:11.095433 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:11.095498 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:11.095599 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:01:11.095979 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.096749 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.098001 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:11.098334 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:11.098425 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:11.098933 (+ 508us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:01:11.099434 (+ 501us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:01:11.099543 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:11.099966 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 19:01:11.100022 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:01:11.107700 15751 raft_consensus.cc:1261] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 5a17c6cf91b943708621f99cb14f6881 [term 1 FOLLOWER]: Refusing update from remote peer 025e8938767147a0b399e29242bd757a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:11.108474 16360 consensus_queue.cc:1035] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [LEADER]: Connected to new peer: Peer: permanent_uuid: "5a17c6cf91b943708621f99cb14f6881" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 41679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:11.113754 15978 raft_consensus.cc:1261] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 1 FOLLOWER]: Refusing update from remote peer 025e8938767147a0b399e29242bd757a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:11.115723 16360 consensus_queue.cc:1035] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [LEADER]: Connected to new peer: Peer: permanent_uuid: "84e43b1f07714866a693dcbe6d5517c6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 42245 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:11.119050 16365 mvcc.cc:205] Tried to move back new op lower bound from 7317047177648340992 to 7317047177425182720. Current Snapshot: MvccSnapshot[applied={T|T < 7317047177648340992}]
I20260810 19:01:11.125326 16367 mvcc.cc:205] Tried to move back new op lower bound from 7317047177648340992 to 7317047177425182720. Current Snapshot: MvccSnapshot[applied={T|T < 7317047177648340992}]
I20260810 19:01:11.129489 16370 mvcc.cc:205] Tried to move back new op lower bound from 7317047177648340992 to 7317047177425182720. Current Snapshot: MvccSnapshot[applied={T|T < 7317047177648340992}]
I20260810 19:01:11.134297 15499 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:11.134464 15499 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49678:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:11.137279 15499 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5623b0878f524952849306d8fe29c8db: Sending DeleteTablet for 3 replicas of tablet 6cd93e93540d4c28b2ca5cb342f5baa0
I20260810 19:01:11.137992 15941 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd93e93540d4c28b2ca5cb342f5baa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:11 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:43724
I20260810 19:01:11.138190 15724 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd93e93540d4c28b2ca5cb342f5baa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:11 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:55660
I20260810 19:01:11.139215 16139 tablet_service.cc:1514] Processing DeleteTablet for tablet 6cd93e93540d4c28b2ca5cb342f5baa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:11 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:53950
I20260810 19:01:11.140096 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 15563
I20260810 19:01:11.158643 16379 tablet_replica.cc:330] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: stopping tablet replica
I20260810 19:01:11.159124 16379 raft_consensus.cc:2229] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:11.167342 16379 raft_consensus.cc:2258] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:11.167680 16385 tablet_replica.cc:330] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: stopping tablet replica
I20260810 19:01:11.168630 16385 raft_consensus.cc:2229] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:01:11.169080 16385 raft_consensus.cc:2258] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:11.170019 16379 ts_tablet_manager.cc:1905] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:11.170632 16385 ts_tablet_manager.cc:1905] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:11.173033 16379 ts_tablet_manager.cc:1918] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:11.173204 16379 log.cc:1196] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/6cd93e93540d4c28b2ca5cb342f5baa0
I20260810 19:01:11.173548 16385 ts_tablet_manager.cc:1918] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:11.173657 16385 log.cc:1196] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/6cd93e93540d4c28b2ca5cb342f5baa0
I20260810 19:01:11.173725 16379 ts_tablet_manager.cc:1939] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 84e43b1f07714866a693dcbe6d5517c6: Deleting consensus metadata
I20260810 19:01:11.174052 16385 ts_tablet_manager.cc:1939] T 6cd93e93540d4c28b2ca5cb342f5baa0 P 025e8938767147a0b399e29242bd757a: Deleting consensus metadata
I20260810 19:01:11.175168 15479 catalog_manager.cc:4841] TS 84e43b1f07714866a693dcbe6d5517c6 (127.27.147.130:42245): tablet 6cd93e93540d4c28b2ca5cb342f5baa0 (table test-table [id=886e8a1f64474ae4b493121964031dff]) successfully deleted
I20260810 19:01:11.175168 15476 catalog_manager.cc:4841] TS 025e8938767147a0b399e29242bd757a (127.27.147.131:40909): tablet 6cd93e93540d4c28b2ca5cb342f5baa0 (table test-table [id=886e8a1f64474ae4b493121964031dff]) successfully deleted
W20260810 19:01:11.175686 15480 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:41679 (error 108)
I20260810 19:01:11.176676 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 15841
I20260810 19:01:11.176798 16319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:11.176425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:41679 (local address 127.0.0.1:55662)
0810 19:01:11.176613 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:11.176689 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260810 19:01:11.176961 15480 catalog_manager.cc:4584] TS 5a17c6cf91b943708621f99cb14f6881 (127.27.147.129:41679): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6cd93e93540d4c28b2ca5cb342f5baa0: Network error: Client connection negotiation failed: client connection to 127.27.147.129:41679: connect: Connection refused (error 111)
I20260810 19:01:11.203920 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 16063
I20260810 19:01:11.217002 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 15442
I20260810 19:01:11.232034 28238 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 19:01:11.577554 28238 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 19:01:11.577662 28238 mini_ranger.cc:66] Stopping Ranger...
I20260810 19:01:11.932036 28238 mini_ranger.cc:68] Stopped Ranger
2026-08-10 19:01:11.932 UTC [13898] LOG: received smart shutdown request
2026-08-10 19:01:11.936 UTC [13898] LOG: background worker "logical replication launcher" (PID 13909) exited with exit code 1
2026-08-10 19:01:11.936 UTC [13902] LOG: shutting down
2026-08-10 19:01:12.098 UTC [13898] LOG: database system is shut down
2026-08-10T19:01:12Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (49204 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:01:12 dist-test-slave-znh6 krb5kdc[16406](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:01:12 dist-test-slave-znh6 krb5kdc[16406](info): set up 2 sockets
Aug 10 19:01:12 dist-test-slave-znh6 krb5kdc[16406](info): commencing operation
krb5kdc: starting...
W20260810 19:01:13.472954 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:01:13 dist-test-slave-znh6 krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:01:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:01:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:01:13.600394 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35833
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:35833,127.27.147.189:44313,127.27.147.188:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:13.710922 16425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:13.711205 16425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:13.711284 16425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:13.715473 16425 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:13.715583 16425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:13.715626 16425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:13.715687 16425 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:13.715728 16425 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:13.720943 16425 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:35833,127.27.147.189:44313,127.27.147.188:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16425
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:13.721994 16425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:13.723037 16425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:13.731599 16432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:13.731823 16437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:13.731966 16425 server_base.cc:1034] running on GCE node
W20260810 19:01:13.732021 16433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:13.732556 16425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:13.733559 16425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:13.734724 16425 hybrid_clock.cc:648] HybridClock initialized: now 1786388473734699 us; error 37 us; skew 500 ppm
Aug 10 19:01:13 dist-test-slave-znh6 krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:13.737414 16425 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:13.738313 16425 webserver.cc:460] Webserver started at http://127.27.147.190:33669/ using document root <none> and password file <none>
I20260810 19:01:13.738917 16425 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:13.739022 16425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:13.739264 16425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:13.741262 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "9093bb0a6968424abe67858b296c280d"
format_stamp: "Formatted at 2026-08-10 19:01:13 on dist-test-slave-znh6"
server_key: "8e6e3ca499c3cd40d1969071cb81d783"
server_key_iv: "f91acf7938ddfa7dab6d4cf6096a46cd"
server_key_version: "encryptionkey@0"
I20260810 19:01:13.741818 16425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "9093bb0a6968424abe67858b296c280d"
format_stamp: "Formatted at 2026-08-10 19:01:13 on dist-test-slave-znh6"
server_key: "8e6e3ca499c3cd40d1969071cb81d783"
server_key_iv: "f91acf7938ddfa7dab6d4cf6096a46cd"
server_key_version: "encryptionkey@0"
I20260810 19:01:13.745558 16425 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 19:01:13.747895 16447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:13.749004 16425 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:01:13.749147 16425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "9093bb0a6968424abe67858b296c280d"
format_stamp: "Formatted at 2026-08-10 19:01:13 on dist-test-slave-znh6"
server_key: "8e6e3ca499c3cd40d1969071cb81d783"
server_key_iv: "f91acf7938ddfa7dab6d4cf6096a46cd"
server_key_version: "encryptionkey@0"
I20260810 19:01:13.749256 16425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:13.764937 16425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:13.767720 16425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:13.767933 16425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:13.775045 16425 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:35833
I20260810 19:01:13.775345 16532 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:35833 every 8 connection(s)
I20260810 19:01:13.775996 16425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:13.778648 16533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:13.782166 16533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:01:13.786273 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 16425
I20260810 19:01:13.786407 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:01:13.786726 28238 external_mini_cluster.cc:1428] Setting key a444168eb3e9e76afbbcba5be1abfda9
Aug 10 19:01:13 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.189@KRBTEST.COM; defaulting to no policy
I20260810 19:01:13.797416 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.784155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35833 (local address 127.0.0.1:46774)
0810 19:01:13.784570 (+ 415us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.784609 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:13.785311 (+ 702us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:13.785918 (+ 607us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:13.785923 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:13.786367 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:13.786723 (+ 356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:13.786744 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:13.787931 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:13.787937 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:13.789096 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:13.789106 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:13.790019 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:13.790022 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:13.790098 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:13.790638 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:13.790654 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:13.792371 (+ 1717us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:13.794689 (+ 2318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:13.794697 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:13.794705 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:13.794986 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:13.795368 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:13.795370 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:13.795372 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:13.795489 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:13.796203 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:13.796208 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:13.796317 (+ 109us) client_negotiation.cc:769] Sending connection context
0810 19:01:13.796457 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 19:01:13.796669 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":100,"threads_started":1}
I20260810 19:01:13.797418 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.784836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46774 (local address 127.27.147.190:35833)
0810 19:01:13.785109 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:13.785117 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:13.785153 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:01:13.785561 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:13.785565 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:13.785732 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:13.785874 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:13.786919 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:13.787785 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:13.789217 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:13.789868 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:13.790053 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:13.792548 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:13.792558 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:13.792573 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:13.792603 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:13.794576 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:13.795104 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:13.795111 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:13.795115 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:13.795263 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:13.795597 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:13.795601 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:13.795602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:13.796087 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:13.796246 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:13.796619 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 19:01:13.796771 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"spinlock_wait_cycles":2816,"thread_start_us":93,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.189@KRBTEST.COM" created.
I20260810 19:01:13.799789 16533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:01:13.800559 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.800250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51802)
0810 19:01:13.800390 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.800478 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 19:01:13.800765 16452 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
I20260810 19:01:13.801563 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.801355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51812)
0810 19:01:13.801461 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.801515 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:01:13.801775 16533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:44313: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab.
Entry for principal kudu/127.27.147.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab.
I20260810 19:01:13.815120 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:44313
--webserver_interface=127.27.147.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab
--principal=kudu/127.27.147.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:35833,127.27.147.189:44313,127.27.147.188:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:13.851029 16533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } attempt: 1
I20260810 19:01:13.852042 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.851748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51818)
0810 19:01:13.851909 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.851968 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 19:01:13.852837 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.852556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51834)
0810 19:01:13.852671 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.852756 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:01:13.852964 16533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:44313: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
I20260810 19:01:13.921121 16533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } attempt: 2
I20260810 19:01:13.922027 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.921709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51838)
0810 19:01:13.921887 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.921948 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 19:01:13.922750 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.922539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51854)
0810 19:01:13.922645 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.922694 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:01:13.922907 16533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:44313: Network error: Client connection negotiation failed: client connection to 127.27.147.189:44313: connect: Connection refused (error 111)
W20260810 19:01:13.925757 16547 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:13.926035 16547 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:13.926134 16547 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:13.930111 16547 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:13.930183 16547 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:13.930233 16547 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:13.930272 16547 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:13.930307 16547 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:13.935992 16547 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:35833,127.27.147.189:44313,127.27.147.188:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab
--principal=kudu/127.27.147.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:44313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.147.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16547
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:13.936966 16547 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:13.937983 16547 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:13.946573 16556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:13.946576 16559 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:13.946606 16555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:13.946910 16547 server_base.cc:1034] running on GCE node
I20260810 19:01:13.947479 16547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:13.948503 16547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:13.949671 16547 hybrid_clock.cc:648] HybridClock initialized: now 1786388473949656 us; error 24 us; skew 500 ppm
Aug 10 19:01:13 dist-test-slave-znh6 krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:13.952464 16547 init.cc:377] Logged in from keytab as kudu/127.27.147.189@KRBTEST.COM (short username kudu)
I20260810 19:01:13.953423 16547 webserver.cc:460] Webserver started at http://127.27.147.189:41427/ using document root <none> and password file <none>
I20260810 19:01:13.953970 16547 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:13.954061 16547 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:13.954279 16547 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:13.956262 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data/instance:
uuid: "5c30f1b633274131bd4794ecf0ee947c"
format_stamp: "Formatted at 2026-08-10 19:01:13 on dist-test-slave-znh6"
server_key: "1a9ee1095ae955f6fcec81163ff8c0c9"
server_key_iv: "3febea981b3900f46c7729f6dec634cd"
server_key_version: "encryptionkey@0"
I20260810 19:01:13.956827 16547 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/wal/instance:
uuid: "5c30f1b633274131bd4794ecf0ee947c"
format_stamp: "Formatted at 2026-08-10 19:01:13 on dist-test-slave-znh6"
server_key: "1a9ee1095ae955f6fcec81163ff8c0c9"
server_key_iv: "3febea981b3900f46c7729f6dec634cd"
server_key_version: "encryptionkey@0"
I20260810 19:01:13.960592 16547 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:01:13.963815 16572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:13.964921 16547 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:01:13.965055 16547 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/wal
uuid: "5c30f1b633274131bd4794ecf0ee947c"
format_stamp: "Formatted at 2026-08-10 19:01:13 on dist-test-slave-znh6"
server_key: "1a9ee1095ae955f6fcec81163ff8c0c9"
server_key_iv: "3febea981b3900f46c7729f6dec634cd"
server_key_version: "encryptionkey@0"
I20260810 19:01:13.965168 16547 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:13.977851 16547 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:13.980513 16547 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:13.980695 16547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:13.987424 16547 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.189:44313
I20260810 19:01:13.987740 16666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.189:44313 every 8 connection(s)
I20260810 19:01:13.988323 16547 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
I20260810 19:01:13.990947 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 16547
I20260810 19:01:13.991084 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-1/wal/instance
I20260810 19:01:13.991407 28238 external_mini_cluster.cc:1428] Setting key 30b4cb2370c37fdcd6c6ab3c15d2eae3
I20260810 19:01:13.991647 16667 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:13.995779 16667 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.147.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.188@KRBTEST.COM" created.
Aug 10 19:01:13 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:14.009693 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.997683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46788 (local address 127.27.147.190:35833)
0810 19:01:13.997816 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:13.997820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:13.998807 (+ 987us) server_negotiation.cc:407] Connection header received
0810 19:01:13.999739 (+ 932us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:13.999742 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:13.999809 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:13.999930 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.001049 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.001668 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.002806 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.003145 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.003274 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.005714 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.005721 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.005727 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.005762 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.007539 (+ 1777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.008025 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.008033 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.008037 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.008136 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.008490 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.008496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.008501 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.008766 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.008885 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.009344 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.009483 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:01:14.010324 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:13.998165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35833 (local address 127.0.0.1:46788)
0810 19:01:13.998658 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:13.998698 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:13.999531 (+ 833us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.000086 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.000093 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.000439 (+ 346us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.000900 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.000912 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.001812 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.001814 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.002664 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.002673 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.003309 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.003313 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.003487 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.004073 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.004089 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.005544 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.007660 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.007665 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.007675 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.007919 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.008241 (+ 322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.008243 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.008245 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.008391 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.008883 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.008888 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.009102 (+ 214us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.009249 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.009500 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":154,"threads_started":1}
I20260810 19:01:14.011816 16667 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:01:14.014060 16533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } attempt: 3
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab.
Entry for principal kudu/127.27.147.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab.
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
I20260810 19:01:14.020198 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:36513
--webserver_interface=127.27.147.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab
--principal=kudu/127.27.147.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:35833,127.27.147.189:44313,127.27.147.188:36513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:14.024034 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.012269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51860)
0810 19:01:14.012417 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.012430 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:14.012538 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.012981 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.012983 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.013170 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.013262 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.013267 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.014517 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.014521 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.015316 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.015324 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.015984 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.015988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.016084 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.016572 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.016592 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.017931 (+ 1339us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.020322 (+ 2391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.020327 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.020332 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.020613 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.021492 (+ 879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.021498 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.021501 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.021614 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.023383 (+ 1769us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.023389 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.023508 (+ 119us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.023631 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.023781 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"spinlock_wait_cycles":31488}
I20260810 19:01:14.024277 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.014668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51872 (local address 127.27.147.189:44313)
0810 19:01:14.015009 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.015018 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.015031 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:01:14.015062 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.015065 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.015100 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.015186 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.015609 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.016319 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.017667 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.018097 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.018171 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.020262 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.020265 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.020268 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.020300 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.022507 (+ 2207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.022978 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.022982 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.022984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.023068 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.023372 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.023376 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.023378 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.023544 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.023683 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.023955 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.024089 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":141,"threads_started":1}
I20260810 19:01:14.024686 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.012380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.27.147.189:44313)
0810 19:01:14.012670 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.012680 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.012698 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:01:14.012731 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.012735 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.012867 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.012972 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.013374 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.014386 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.015428 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.015867 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.015990 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.018073 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.018082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.018091 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.018112 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.020197 (+ 2085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.020739 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.020748 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.020753 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.021397 (+ 644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.022910 (+ 1513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.022913 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.022915 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.023238 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.024242 (+ 1004us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.024325 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.024465 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":115,"threads_started":1}
I20260810 19:01:14.024870 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.014488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51872)
0810 19:01:14.014591 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.014606 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:14.014676 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.015220 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.015222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.015349 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.015496 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.015502 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.016433 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.016438 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.017532 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.017543 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.018202 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.018208 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.018322 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.018840 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.018859 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.020119 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.022624 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.022629 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.022631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.022888 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.023172 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.023175 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.023177 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.023272 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.023645 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.023648 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.023740 (+ 92us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.023869 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.024003 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 19:01:14.026484 16533 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:01:14.026674 16667 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:01:14.027241 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.026944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57524)
0810 19:01:14.027091 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.027159 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:01:14.027336 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.027089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57534)
0810 19:01:14.027181 (+ 92us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.027255 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260810 19:01:14.027524 16579 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
I20260810 19:01:14.028439 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.028200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57550)
0810 19:01:14.028340 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.028392 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 19:01:14.028540 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.028200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57548)
0810 19:01:14.028386 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.028461 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
W20260810 19:01:14.028677 16533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:36513: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
W20260810 19:01:14.028661 16667 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:36513: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
I20260810 19:01:14.059839 16533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } attempt: 1
I20260810 19:01:14.060678 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.060389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57556)
0810 19:01:14.060540 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.060600 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 19:01:14.061484 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.061270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57566)
0810 19:01:14.061381 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.061427 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:01:14.061707 16533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:36513: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
I20260810 19:01:14.077911 16667 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } attempt: 1
I20260810 19:01:14.078697 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.078404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57582)
0810 19:01:14.078561 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.078624 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 19:01:14.079418 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.079205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57592)
0810 19:01:14.079314 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.079368 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:01:14.079547 16667 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:36513: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
W20260810 19:01:14.136112 16677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:14.136407 16677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:14.136487 16677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 19:01:14.136866 16533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } attempt: 2
I20260810 19:01:14.137815 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.137516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57604)
0810 19:01:14.137671 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.137734 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 19:01:14.138574 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.138352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57614)
0810 19:01:14.138467 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.138519 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:01:14.138705 16533 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:36513: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
W20260810 19:01:14.140897 16677 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:14.140970 16677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:14.141017 16677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:14.141070 16677 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:14.141108 16677 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:14.146306 16677 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:35833,127.27.147.189:44313,127.27.147.188:36513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab
--principal=kudu/127.27.147.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:36513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.147.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:14.147439 16677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:14.147719 16667 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } attempt: 2
I20260810 19:01:14.148465 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.148199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57628)
0810 19:01:14.148333 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.148398 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:01:14.148519 16677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:01:14.149294 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.149048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57634)
0810 19:01:14.149181 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.149234 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260810 19:01:14.149430 16667 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:36513: Network error: Client connection negotiation failed: client connection to 127.27.147.188:36513: connect: Connection refused (error 111)
W20260810 19:01:14.157562 16685 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:14.157553 16690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:14.157553 16687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:14.157603 16677 server_base.cc:1034] running on GCE node
I20260810 19:01:14.158310 16677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:14.159441 16677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:14.160650 16677 hybrid_clock.cc:648] HybridClock initialized: now 1786388474160629 us; error 32 us; skew 500 ppm
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388474, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:14.163566 16677 init.cc:377] Logged in from keytab as kudu/127.27.147.188@KRBTEST.COM (short username kudu)
I20260810 19:01:14.164413 16677 webserver.cc:460] Webserver started at http://127.27.147.188:43519/ using document root <none> and password file <none>
I20260810 19:01:14.164932 16677 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:14.164989 16677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:14.165150 16677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:14.167240 16677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data/instance:
uuid: "c9e180a7abf842da88c88a074b69c7ed"
format_stamp: "Formatted at 2026-08-10 19:01:14 on dist-test-slave-znh6"
server_key: "5cff367b14efce0830280d47ec537e0b"
server_key_iv: "b30e9ab26a23a0f18cee503607c54661"
server_key_version: "encryptionkey@0"
I20260810 19:01:14.167847 16677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/wal/instance:
uuid: "c9e180a7abf842da88c88a074b69c7ed"
format_stamp: "Formatted at 2026-08-10 19:01:14 on dist-test-slave-znh6"
server_key: "5cff367b14efce0830280d47ec537e0b"
server_key_iv: "b30e9ab26a23a0f18cee503607c54661"
server_key_version: "encryptionkey@0"
I20260810 19:01:14.171947 16677 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:01:14.174281 16702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:14.175563 16677 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:01:14.175693 16677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/wal
uuid: "c9e180a7abf842da88c88a074b69c7ed"
format_stamp: "Formatted at 2026-08-10 19:01:14 on dist-test-slave-znh6"
server_key: "5cff367b14efce0830280d47ec537e0b"
server_key_iv: "b30e9ab26a23a0f18cee503607c54661"
server_key_version: "encryptionkey@0"
I20260810 19:01:14.175796 16677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:14.197741 16677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:14.200959 16677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:14.201159 16677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:14.208123 16677 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.188:36513
I20260810 19:01:14.208195 16805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.188:36513 every 8 connection(s)
I20260810 19:01:14.209059 16677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
I20260810 19:01:14.211743 16806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:14.215474 16806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:01:14.217414 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 16677
I20260810 19:01:14.217559 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786388305279399-28238-0/minicluster-data/master-2/wal/instance
I20260810 19:01:14.217844 28238 external_mini_cluster.cc:1428] Setting key 76d51c513ec5e4221a02276dc6795421
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388474, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:01:14.229079 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.217290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46798 (local address 127.27.147.190:35833)
0810 19:01:14.217454 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.217460 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.218274 (+ 814us) server_negotiation.cc:407] Connection header received
0810 19:01:14.219422 (+ 1148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.219426 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.219482 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.219595 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.220369 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.220879 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.221865 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.222082 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.222191 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.224868 (+ 2677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.224876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.224878 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.224903 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.226583 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.227168 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.227173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.227177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.227267 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.227950 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.227953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.227954 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.228219 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.228330 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.228771 (+ 441us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.228914 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:01:14.229878 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.217654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35833 (local address 127.0.0.1:46798)
0810 19:01:14.218112 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.218162 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:14.219222 (+ 1060us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.219643 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.219649 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.219992 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.220218 (+ 226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.220230 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.220992 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.220996 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.221731 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.221739 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.222237 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.222241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.222451 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.223047 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.223071 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.224714 (+ 1643us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.226693 (+ 1979us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.226702 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.226709 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.227065 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.227656 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.227659 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.227660 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.227841 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.228338 (+ 497us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.228343 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.228509 (+ 166us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.228681 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.228992 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"mutex_wait_us":5,"spinlock_wait_cycles":13056,"thread_start_us":124,"threads_started":1}
I20260810 19:01:14.229879 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.218510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57646 (local address 127.27.147.188:36513)
0810 19:01:14.218837 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.218850 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.219041 (+ 191us) server_negotiation.cc:407] Connection header received
0810 19:01:14.219230 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.219243 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.219414 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.219616 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.220450 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.221290 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.222127 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.222527 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.222648 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.225289 (+ 2641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.225294 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.225327 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.225357 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.227260 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.227647 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.227652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.227653 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.227761 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.228068 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.228071 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.228073 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.228525 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.228708 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.228954 (+ 246us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.229082 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":181,"spinlock_wait_cycles":10624,"thread_start_us":82,"threads_started":1}
I20260810 19:01:14.231513 16806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388474, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:14.237860 16533 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } attempt: 3
I20260810 19:01:14.240589 16667 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } attempt: 3
I20260810 19:01:14.240942 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.232267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51886 (local address 127.27.147.189:44313)
0810 19:01:14.232408 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.232414 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.232429 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:01:14.232475 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.232478 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.232521 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.232626 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.233160 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.233773 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.234584 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.234974 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.235057 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.237160 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.237163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.237165 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.237188 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.238907 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.239405 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.239409 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.239411 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.239502 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.239816 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.239822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.239827 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.240077 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.240191 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.240625 (+ 434us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.240749 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:01:14.241604 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.232118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:44313 (local address 127.0.0.1:51886)
0810 19:01:14.232230 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.232245 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:14.232318 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.232750 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.232753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.232925 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.233026 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.233032 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.233888 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.233891 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.234467 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.234476 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.235082 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.235086 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.235193 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.235606 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.235617 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.237006 (+ 1389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.239029 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.239036 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.239039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.239294 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.239610 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.239613 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.239615 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.239705 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.240379 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.240382 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.240459 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.240569 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.240755 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 19:01:14.241740 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.232531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46812 (local address 127.27.147.190:35833)
0810 19:01:14.232657 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.232661 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.232673 (+ 12us) server_negotiation.cc:407] Connection header received
0810 19:01:14.232709 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.232712 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.232749 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.232826 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.233459 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.233973 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.234770 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.235075 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.235176 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.237840 (+ 2664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.237844 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.237846 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.237875 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.239531 (+ 1656us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.239983 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.239986 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.239988 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.240072 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.240365 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.240369 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.240370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.240568 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.240806 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.241486 (+ 680us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.241595 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:01:14.244565 16806 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:01:14.249650 16537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.238565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57652)
0810 19:01:14.238906 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.238918 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:14.239002 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.239509 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.239512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.239700 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.239813 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.239819 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.240973 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.240976 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.241712 (+ 736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.241722 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.242394 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.242399 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.242492 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.243025 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.243045 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.244422 (+ 1377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.247849 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.247854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.247856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.248111 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.248436 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.248439 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.248441 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.248531 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.249102 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.249107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.249222 (+ 115us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.249329 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.249481 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388473, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
I20260810 19:01:14.250546 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.239138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57652 (local address 127.27.147.188:36513)
0810 19:01:14.239279 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.239284 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.239300 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:01:14.239337 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.239340 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.239382 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.239502 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.239899 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.240853 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.241875 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.242252 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.243614 (+ 1362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.244931 (+ 1317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.244937 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.244943 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.244972 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.247740 (+ 2768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.248242 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.248247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.248250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.248348 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.248738 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.248741 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.248744 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.248966 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.250185 (+ 1219us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.250261 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.250381 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
Aug 10 19:01:14 dist-test-slave-znh6 krb5kdc[16406](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388474, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:01:14.251425 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.240971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57658)
0810 19:01:14.241218 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.241229 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:14.241285 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.241946 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.241949 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.242122 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.242240 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.242247 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.243714 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.243717 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.244360 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.244370 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.245053 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.245057 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.245144 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.245527 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.245545 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.247032 (+ 1487us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.249422 (+ 2390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.249425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.249427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.249669 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.250119 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.250122 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.250124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.250208 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.250897 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.250900 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.250994 (+ 94us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.251093 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.251226 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"spinlock_wait_cycles":1664}
I20260810 19:01:14.252146 16815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.241222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57658 (local address 127.27.147.188:36513)
0810 19:01:14.241672 (+ 450us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.241691 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.241707 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:01:14.241747 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.241751 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.241786 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.242460 (+ 674us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.242556 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.243583 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.244523 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.244913 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.245331 (+ 418us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.247185 (+ 1854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.247188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.247191 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.247215 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.249293 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.249909 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.249912 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.249913 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.250002 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.250314 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.250318 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.250322 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.250738 (+ 416us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.251698 (+ 960us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.251780 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.251943 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":125,"threads_started":1}
I20260810 19:01:14.255211 16810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.245051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:36513 (local address 127.0.0.1:57666)
0810 19:01:14.245513 (+ 462us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:14.245526 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:14.245596 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:14.246470 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:14.246473 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:14.246627 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:14.246722 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.246732 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.247954 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.247956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.248738 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:14.248752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.249476 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.249480 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:14.249564 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.249968 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:14.249981 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:14.251814 (+ 1833us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:14.253674 (+ 1860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.253680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.253683 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.253943 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.254222 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:14.254225 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:14.254226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:14.254305 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:14.254656 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:14.254659 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:14.254738 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 19:01:14.254893 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 19:01:14.255023 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401}
I20260810 19:01:14.255374 16816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.245220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.27.147.188:36513)
0810 19:01:14.246162 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.246171 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.246184 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:01:14.246215 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.246218 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.246255 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.246343 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.246861 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.247825 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.248877 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.249263 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.249366 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.251953 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.251956 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.251959 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.251980 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.253571 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.254038 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.254043 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.254045 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.254124 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.254400 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.254405 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.254407 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.254561 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.254664 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.254939 (+ 275us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.255078 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":860,"thread_start_us":179,"threads_started":1}
I20260810 19:01:14.255496 16533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: Bootstrap starting.
I20260810 19:01:14.255688 16667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: Bootstrap starting.
I20260810 19:01:14.258435 16533 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:14.258451 16667 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:14.259418 16667 log.cc:824] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:14.259418 16533 log.cc:824] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:14.261225 16806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed: Bootstrap starting.
I20260810 19:01:14.261394 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.243616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51900 (local address 127.27.147.189:44313)
0810 19:01:14.243723 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.243727 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.244473 (+ 746us) server_negotiation.cc:407] Connection header received
0810 19:01:14.244585 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.244588 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.244626 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.244690 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.245471 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.246303 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.247550 (+ 1247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.247869 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.247969 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.250975 (+ 3006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.250979 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.250981 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.251008 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.252696 (+ 1688us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.253280 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.253284 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.253289 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.253378 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.260096 (+ 6718us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.260100 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.260101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.260329 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.260517 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.261139 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.261258 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:01:14.261838 16667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: No bootstrap required, opened a new log
I20260810 19:01:14.263991 16806 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:14.264338 16533 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: No bootstrap required, opened a new log
I20260810 19:01:14.264704 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:01:14.264878 16806 log.cc:824] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:14.265717 16667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.265921 16667 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:14.265970 16667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c30f1b633274131bd4794ecf0ee947c, State: Initialized, Role: FOLLOWER
I20260810 19:01:14.266434 16667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.267781 16667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:14.268734 16533 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.269002 16533 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:14.269060 16533 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9093bb0a6968424abe67858b296c280d, State: Initialized, Role: FOLLOWER
I20260810 19:01:14.269556 16533 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.270998 16533 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [sys.catalog]: configured and running, proceeding with master startup.
W20260810 19:01:14.277387 16857 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:01:14.277467 16857 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260810 19:01:14.277974 16873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:01:14.278061 16873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:01:14.278702 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.267432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51910 (local address 127.27.147.189:44313)
0810 19:01:14.267538 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.267542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.267721 (+ 179us) server_negotiation.cc:407] Connection header received
0810 19:01:14.267955 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.267959 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.268011 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.268064 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.269177 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.270077 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.271316 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.271628 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.271691 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.274677 (+ 2986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.274681 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.274683 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.274708 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.276347 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.277152 (+ 805us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.277155 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.277157 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.277240 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.277651 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.277655 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.277656 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.277872 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.277971 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.278431 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.278542 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:01:14.278870 16806 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed: No bootstrap required, opened a new log
I20260810 19:01:14.279374 16811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.267244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57680 (local address 127.27.147.188:36513)
0810 19:01:14.267476 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.267483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.268007 (+ 524us) server_negotiation.cc:407] Connection header received
0810 19:01:14.268171 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.268176 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.268240 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.268298 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.269175 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.269934 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.271255 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.271592 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.271673 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.273712 (+ 2039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.273715 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.273718 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.273744 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.275527 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.276809 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.276814 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.276815 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.276900 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.278378 (+ 1478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.278382 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.278385 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.278584 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.278678 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.279090 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.279222 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260810 19:01:14.279207 16425 master_runner.cc:186] Error getting master registration for 127.27.147.188:36513: OK, instance_id { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" instance_seqno: 1786388474206186 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:01:14.283578 16425 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:01:14.283517 16806 raft_consensus.cc:348] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.283743 16806 raft_consensus.cc:374] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:14.283807 16806 raft_consensus.cc:729] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c9e180a7abf842da88c88a074b69c7ed, State: Initialized, Role: FOLLOWER
I20260810 19:01:14.284204 16806 consensus_queue.cc:260] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.284708 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.284931 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:01:14.285540 16806 sys_catalog.cc:564] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:14.273249 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.286979 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [sys.catalog]: This master's current role is: FOLLOWER
W20260810 19:01:14.290437 16896 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:01:14.290519 16896 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:01:14.290769 16831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.291003 16831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:01:14.297551 16677 master_runner.cc:263] No leader master found from master c9e180a7abf842da88c88a074b69c7ed
I20260810 19:01:14.297613 16677 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:01:14.298362 16547 master_runner.cc:263] No leader master found from master 5c30f1b633274131bd4794ecf0ee947c
I20260810 19:01:14.298435 16547 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:01:14.300249 16538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:14.266991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46820 (local address 127.27.147.190:35833)
0810 19:01:14.270163 (+ 3172us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:14.270168 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:14.270182 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:01:14.270216 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:14.270219 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:14.270258 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:14.270373 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:14.271333 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.272074 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.284344 (+ 12270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:14.284660 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:14.284758 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:14.286479 (+ 1721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:14.286485 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:14.286487 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:14.286516 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:14.297245 (+ 10729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.298921 (+ 1676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.298925 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.298927 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.299012 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:14.299315 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:14.299318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:14.299320 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:14.299536 (+ 216us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:14.299640 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:14.299982 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 19:01:14.300101 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3098}
I20260810 19:01:14.327054 16876 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:14.327234 16876 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.328226 16876 leader_election.cc:288] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9093bb0a6968424abe67858b296c280d (127.27.147.190:35833), 5c30f1b633274131bd4794ecf0ee947c (127.27.147.189:44313)
I20260810 19:01:14.329123 16618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9e180a7abf842da88c88a074b69c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c30f1b633274131bd4794ecf0ee947c" is_pre_election: true
I20260810 19:01:14.329435 16618 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e180a7abf842da88c88a074b69c7ed in term 0.
I20260810 19:01:14.329967 16489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9e180a7abf842da88c88a074b69c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9093bb0a6968424abe67858b296c280d" is_pre_election: true
I20260810 19:01:14.330226 16489 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c9e180a7abf842da88c88a074b69c7ed in term 0.
I20260810 19:01:14.330614 16704 leader_election.cc:302] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9093bb0a6968424abe67858b296c280d, c9e180a7abf842da88c88a074b69c7ed; no voters:
I20260810 19:01:14.331096 16876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:01:14.331184 16876 raft_consensus.cc:482] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:01:14.331238 16876 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:14.332574 16876 raft_consensus.cc:504] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.332890 16876 leader_election.cc:288] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [CANDIDATE]: Term 1 election: Requested vote from peers 9093bb0a6968424abe67858b296c280d (127.27.147.190:35833), 5c30f1b633274131bd4794ecf0ee947c (127.27.147.189:44313)
I20260810 19:01:14.333308 16618 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9e180a7abf842da88c88a074b69c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5c30f1b633274131bd4794ecf0ee947c"
I20260810 19:01:14.333423 16618 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:14.334681 16618 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e180a7abf842da88c88a074b69c7ed in term 1.
I20260810 19:01:14.335103 16489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c9e180a7abf842da88c88a074b69c7ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9093bb0a6968424abe67858b296c280d"
I20260810 19:01:14.335222 16489 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:14.336469 16489 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c9e180a7abf842da88c88a074b69c7ed in term 1.
I20260810 19:01:14.336767 16704 leader_election.cc:302] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9093bb0a6968424abe67858b296c280d, c9e180a7abf842da88c88a074b69c7ed; no voters:
I20260810 19:01:14.338757 16876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:14.339042 16876 raft_consensus.cc:686] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [term 1 LEADER]: Becoming Leader. State: Replica: c9e180a7abf842da88c88a074b69c7ed, State: Running, Role: LEADER
I20260810 19:01:14.339432 16876 consensus_queue.cc:237] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } }
I20260810 19:01:14.341354 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e180a7abf842da88c88a074b69c7ed. Latest consensus state: current_term: 1 leader_uuid: "c9e180a7abf842da88c88a074b69c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.341477 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:14.341907 16905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:14.344857 16905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:14.352929 16618 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [term 1 FOLLOWER]: Refusing update from remote peer c9e180a7abf842da88c88a074b69c7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:14.354226 16876 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:14.360998 16831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e180a7abf842da88c88a074b69c7ed. Latest consensus state: current_term: 1 leader_uuid: "c9e180a7abf842da88c88a074b69c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.361129 16831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:01:14.362131 16489 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [term 1 FOLLOWER]: Refusing update from remote peer c9e180a7abf842da88c88a074b69c7ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:14.362731 16900 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:14.366281 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c9e180a7abf842da88c88a074b69c7ed. Latest consensus state: current_term: 1 leader_uuid: "c9e180a7abf842da88c88a074b69c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.366396 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:01:14.375746 16900 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9e180a7abf842da88c88a074b69c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.375870 16900 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:14.377000 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9e180a7abf842da88c88a074b69c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.377112 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:01:14.377492 16906 mvcc.cc:205] Tried to move back new op lower bound from 7317047190941691904 to 7317047190897422336. Current Snapshot: MvccSnapshot[applied={T|T < 7317047190941691904}]
I20260810 19:01:14.377842 16912 mvcc.cc:205] Tried to move back new op lower bound from 7317047190941691904 to 7317047190897422336. Current Snapshot: MvccSnapshot[applied={T|T < 7317047190941691904}]
I20260810 19:01:14.378525 16876 sys_catalog.cc:455] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c9e180a7abf842da88c88a074b69c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.378638 16876 sys_catalog.cc:458] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:14.379721 16831 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c9e180a7abf842da88c88a074b69c7ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9093bb0a6968424abe67858b296c280d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35833 } } peers { permanent_uuid: "5c30f1b633274131bd4794ecf0ee947c" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 44313 } } peers { permanent_uuid: "c9e180a7abf842da88c88a074b69c7ed" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 36513 } } }
I20260810 19:01:14.379863 16831 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:01:14.380574 16907 mvcc.cc:205] Tried to move back new op lower bound from 7317047190941691904 to 7317047190897422336. Current Snapshot: MvccSnapshot[applied={T|T < 7317047190941691904}]
I20260810 19:01:14.387526 16905 catalog_manager.cc:1347] Generated new cluster ID: 1c57e78b7e3d4d869ab0f0934dc475b4
I20260810 19:01:14.387595 16905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:14.424511 16905 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:01:14.425796 16905 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:14.456072 16905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed: Generated new TSK 0
I20260810 19:01:14.457054 16905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:01:15.278887 16857 catalog_manager.cc:1259] Loaded cluster ID: 1c57e78b7e3d4d869ab0f0934dc475b4
I20260810 19:01:15.279013 16857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: loading cluster ID for follower catalog manager: success
I20260810 19:01:15.279335 16873 catalog_manager.cc:1259] Loaded cluster ID: 1c57e78b7e3d4d869ab0f0934dc475b4
I20260810 19:01:15.279390 16873 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: loading cluster ID for follower catalog manager: success
I20260810 19:01:15.280874 16857 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: acquiring CA information for follower catalog manager: success
I20260810 19:01:15.280962 16873 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: acquiring CA information for follower catalog manager: success
I20260810 19:01:15.282047 16857 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5c30f1b633274131bd4794ecf0ee947c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:01:15.282047 16873 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9093bb0a6968424abe67858b296c280d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:01:15.525368 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 16425
W20260810 19:01:15.546330 16704 connection.cc:535] client connection to 127.27.147.190:35833 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 19:01:15.546546 16704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 19:01:15.546645 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 16547
I20260810 19:01:15.552103 16931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:15.551711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35833 (local address 127.0.0.1:57474)
0810 19:01:15.551946 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:15.552011 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:35833: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
W20260810 19:01:15.552294 16704 consensus_peers.cc:487] T 00000000000000000000000000000000 P c9e180a7abf842da88c88a074b69c7ed -> Peer 9093bb0a6968424abe67858b296c280d (127.27.147.190:35833): Couldn't send request to peer 9093bb0a6968424abe67858b296c280d. Status: Network error: Client connection negotiation failed: client connection to 127.27.147.190:35833: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 19:01:15.558203 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 16677
2026-08-10T19:01:15Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3340 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:01:15 dist-test-slave-znh6 krb5kdc[16942](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:01:15 dist-test-slave-znh6 krb5kdc[16942](info): set up 2 sockets
Aug 10 19:01:15 dist-test-slave-znh6 krb5kdc[16942](info): commencing operation
krb5kdc: starting...
W20260810 19:01:16.808812 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.003s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:01:16 dist-test-slave-znh6 krb5kdc[16942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:01:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:01:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:01:16.934345 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37913
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:41277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37913
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:17.047948 16963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:17.048241 16963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:17.048345 16963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:17.052371 16963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:17.052451 16963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:17.052489 16963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:17.052544 16963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:17.052582 16963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:17.057785 16963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:41277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37913
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:17.058854 16963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:17.059867 16963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:17.068389 16978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:17.068397 16976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:17.068406 16971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:17.068439 16963 server_base.cc:1034] running on GCE node
I20260810 19:01:17.069258 16963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:17.070415 16963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:17.071607 16963 hybrid_clock.cc:648] HybridClock initialized: now 1786388477071587 us; error 32 us; skew 500 ppm
Aug 10 19:01:17 dist-test-slave-znh6 krb5kdc[16942](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388477, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:17.074406 16963 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:17.075346 16963 webserver.cc:460] Webserver started at http://127.27.147.190:43663/ using document root <none> and password file <none>
I20260810 19:01:17.075901 16963 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:17.075995 16963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:17.076231 16963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:17.078312 16963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "3001d3e99f314f63938c5ebbb58b5707"
format_stamp: "Formatted at 2026-08-10 19:01:17 on dist-test-slave-znh6"
server_key: "3d5908c9ce79056ee61ddc7fb6e4338b"
server_key_iv: "b3b93c5788a9a4d7749f43b03aad1f92"
server_key_version: "encryptionkey@0"
I20260810 19:01:17.078953 16963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "3001d3e99f314f63938c5ebbb58b5707"
format_stamp: "Formatted at 2026-08-10 19:01:17 on dist-test-slave-znh6"
server_key: "3d5908c9ce79056ee61ddc7fb6e4338b"
server_key_iv: "b3b93c5788a9a4d7749f43b03aad1f92"
server_key_version: "encryptionkey@0"
I20260810 19:01:17.082772 16963 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 19:01:17.085394 16994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:17.086563 16963 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:17.086711 16963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "3001d3e99f314f63938c5ebbb58b5707"
format_stamp: "Formatted at 2026-08-10 19:01:17 on dist-test-slave-znh6"
server_key: "3d5908c9ce79056ee61ddc7fb6e4338b"
server_key_iv: "b3b93c5788a9a4d7749f43b03aad1f92"
server_key_version: "encryptionkey@0"
I20260810 19:01:17.086849 16963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:17.097713 16963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:17.101040 16963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:17.101240 16963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:17.108428 16963 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37913
I20260810 19:01:17.108510 17095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37913 every 8 connection(s)
I20260810 19:01:17.109447 16963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:17.110647 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 16963
I20260810 19:01:17.110796 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:01:17.111111 28238 external_mini_cluster.cc:1428] Setting key 177322e3e4532f44cc37f6559cce19a1
I20260810 19:01:17.112589 17101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:17.119536 17101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707: Bootstrap starting.
Aug 10 19:01:17 dist-test-slave-znh6 krb5kdc[16942](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:17.121966 17101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:17.122773 17101 log.cc:824] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:17.124732 17101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707: No bootstrap required, opened a new log
I20260810 19:01:17.125800 17105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:17.112461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48448 (local address 127.27.147.190:37913)
0810 19:01:17.112955 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:17.112975 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:17.113316 (+ 341us) server_negotiation.cc:407] Connection header received
0810 19:01:17.114071 (+ 755us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:17.114087 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:17.114412 (+ 325us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:17.114693 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:17.115657 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:17.116690 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:17.117855 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:17.118309 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:17.118439 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:17.121232 (+ 2793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:17.121236 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:17.121249 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:17.121277 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:17.123145 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:17.123602 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:17.123607 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:17.123612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:17.123719 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:17.124003 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:17.124007 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:17.124009 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:17.124384 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:17.124584 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:17.124845 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 19:01:17.125129 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":143,"threads_started":1}
I20260810 19:01:17.127802 17101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3001d3e99f314f63938c5ebbb58b5707" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37913 } }
I20260810 19:01:17.128105 17101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:17.128201 17101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3001d3e99f314f63938c5ebbb58b5707, State: Initialized, Role: FOLLOWER
I20260810 19:01:17.128659 17101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3001d3e99f314f63938c5ebbb58b5707" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37913 } }
I20260810 19:01:17.128819 17101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:17.128903 17101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:17.129017 17101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:17.129051 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:01:17.130115 17101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3001d3e99f314f63938c5ebbb58b5707" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37913 } }
I20260810 19:01:17.130455 17101 leader_election.cc:302] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3001d3e99f314f63938c5ebbb58b5707; no voters:
I20260810 19:01:17.130780 17101 leader_election.cc:288] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:01:17.130897 17107 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:17.131151 17107 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [term 1 LEADER]: Becoming Leader. State: Replica: 3001d3e99f314f63938c5ebbb58b5707, State: Running, Role: LEADER
I20260810 19:01:17.131500 17107 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3001d3e99f314f63938c5ebbb58b5707" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37913 } }
I20260810 19:01:17.132009 17101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:17.133261 17111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3001d3e99f314f63938c5ebbb58b5707. Latest consensus state: current_term: 1 leader_uuid: "3001d3e99f314f63938c5ebbb58b5707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3001d3e99f314f63938c5ebbb58b5707" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37913 } } }
I20260810 19:01:17.133301 17110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3001d3e99f314f63938c5ebbb58b5707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3001d3e99f314f63938c5ebbb58b5707" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37913 } } }
I20260810 19:01:17.133440 17111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:17.133455 17110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:17.134361 17119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:17.136940 17119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:17.141628 17119 catalog_manager.cc:1347] Generated new cluster ID: 2390a6d6006c4788a65757dcaaf55439
I20260810 19:01:17.141690 17119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:17.171046 17119 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:01:17.172274 17119 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:17.184146 17119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3001d3e99f314f63938c5ebbb58b5707: Generated new TSK 0
I20260810 19:01:17.184945 17119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:01:17.236459 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 16963
2026-08-10T19:01:17Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1683 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:01:17 dist-test-slave-znh6 krb5kdc[17148](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:01:17 dist-test-slave-znh6 krb5kdc[17148](info): set up 2 sockets
Aug 10 19:01:17 dist-test-slave-znh6 krb5kdc[17148](info): commencing operation
krb5kdc: starting...
W20260810 19:01:18.480882 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.212s user 0.005s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:01:18 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:01:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:01:18.604019 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41613
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:18.715533 17172 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:18.715809 17172 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:18.715893 17172 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:18.719861 17172 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:18.719935 17172 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:18.719973 17172 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:18.720031 17172 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:18.720070 17172 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:18.725179 17172 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17172
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:18.726163 17172 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:18.727166 17172 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:18.735532 17179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:18.735599 17183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:18.735610 17178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:18.735704 17172 server_base.cc:1034] running on GCE node
I20260810 19:01:18.736394 17172 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:18.737542 17172 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:18.738907 17172 hybrid_clock.cc:648] HybridClock initialized: now 1786388478738858 us; error 47 us; skew 500 ppm
Aug 10 19:01:18 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388478, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:18.741652 17172 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:18.742501 17172 webserver.cc:460] Webserver started at http://127.27.147.190:45505/ using document root <none> and password file <none>
I20260810 19:01:18.743108 17172 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:18.743199 17172 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:18.743470 17172 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:18.745504 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "56773a037d81472484f5dc475a2742f8"
format_stamp: "Formatted at 2026-08-10 19:01:18 on dist-test-slave-znh6"
server_key: "b447cfcdbd530d55a2ddd24d9988c66d"
server_key_iv: "78f02d729a73e0d582716ce436d17c0c"
server_key_version: "encryptionkey@0"
I20260810 19:01:18.746099 17172 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "56773a037d81472484f5dc475a2742f8"
format_stamp: "Formatted at 2026-08-10 19:01:18 on dist-test-slave-znh6"
server_key: "b447cfcdbd530d55a2ddd24d9988c66d"
server_key_iv: "78f02d729a73e0d582716ce436d17c0c"
server_key_version: "encryptionkey@0"
I20260810 19:01:18.749804 17172 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 19:01:18.752238 17194 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:18.753425 17172 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:01:18.753567 17172 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "56773a037d81472484f5dc475a2742f8"
format_stamp: "Formatted at 2026-08-10 19:01:18 on dist-test-slave-znh6"
server_key: "b447cfcdbd530d55a2ddd24d9988c66d"
server_key_iv: "78f02d729a73e0d582716ce436d17c0c"
server_key_version: "encryptionkey@0"
I20260810 19:01:18.753674 17172 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:18.771325 17172 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:18.774175 17172 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:18.774379 17172 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:18.781742 17172 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:41613
I20260810 19:01:18.782090 17283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:41613 every 8 connection(s)
I20260810 19:01:18.783056 17172 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:18.785805 17284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:18.789942 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 17172
I20260810 19:01:18.790091 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:01:18.790386 28238 external_mini_cluster.cc:1428] Setting key 9e6de5e79779277f88f7f867b3a2ec47
I20260810 19:01:18.791541 17284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Bootstrap starting.
I20260810 19:01:18.793820 17284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:18.794693 17284 log.cc:824] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:18.796823 17284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: No bootstrap required, opened a new log
Aug 10 19:01:18 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388478, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:18.801693 17284 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:18.801964 17284 raft_consensus.cc:374] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:18.802039 17284 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56773a037d81472484f5dc475a2742f8, State: Initialized, Role: FOLLOWER
I20260810 19:01:18.802565 17284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:18.802736 17284 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:18.802806 17284 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:18.802984 17284 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:18.803850 17284 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:18.804222 17284 leader_election.cc:302] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56773a037d81472484f5dc475a2742f8; no voters:
I20260810 19:01:18.804576 17284 leader_election.cc:288] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:01:18.804641 17293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:18.804872 17293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 LEADER]: Becoming Leader. State: Replica: 56773a037d81472484f5dc475a2742f8, State: Running, Role: LEADER
I20260810 19:01:18.805147 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:18.791776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60646 (local address 127.27.147.190:41613)
0810 19:01:18.792191 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:18.792207 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:18.792242 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:01:18.792860 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:18.792875 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:18.793179 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:18.793450 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:18.794414 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:18.795468 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:18.796575 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:18.797070 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:18.797187 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:18.799956 (+ 2769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:18.799965 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:18.799987 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:18.800027 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:18.801951 (+ 1924us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:18.802476 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:18.802481 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:18.802486 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:18.802630 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:18.803004 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:18.803008 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:18.803010 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:18.803453 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:18.803678 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:18.803956 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 19:01:18.804226 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":274,"thread_start_us":147,"threads_started":1}
I20260810 19:01:18.805301 17293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:18.805824 17284 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:18.806546 17293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56773a037d81472484f5dc475a2742f8. Latest consensus state: current_term: 1 leader_uuid: "56773a037d81472484f5dc475a2742f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } } }
I20260810 19:01:18.807349 17293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:18.807235 17294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "56773a037d81472484f5dc475a2742f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } } }
I20260810 19:01:18.808197 17312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:18.808336 17294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:18.811143 17312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:18.816251 17312 catalog_manager.cc:1347] Generated new cluster ID: 8d79316e09024497982c62942722d8ee
I20260810 19:01:18.816313 17312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:18.841434 17312 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:01:18.842643 17312 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:18.854760 17312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Generated new TSK 0
I20260810 19:01:18.855608 17312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 19:01:18.896277 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41613
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:19.008227 17329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:19.008643 17329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:19.008737 17329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:19.013826 17329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:19.013908 17329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:19.014001 17329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:01:19.019140 17329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17329
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:19.020083 17329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:19.021046 17329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:19.030642 17340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:19.030606 17338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:19.030597 17342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:19.030766 17329 server_base.cc:1034] running on GCE node
I20260810 19:01:19.031276 17329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:19.031842 17329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:19.033023 17329 hybrid_clock.cc:648] HybridClock initialized: now 1786388479033003 us; error 32 us; skew 500 ppm
Aug 10 19:01:19 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:19.035619 17329 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:01:19.036448 17329 webserver.cc:460] Webserver started at http://127.27.147.129:46327/ using document root <none> and password file <none>
I20260810 19:01:19.036983 17329 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:19.037074 17329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:19.037344 17329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:19.039433 17329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "286f8b8e33c4427e886d8b0765bfce65"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "91ec4736df952b00358fc41daa08745c"
server_key_iv: "f215bc9ddd8ab030422916931bca7f08"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.039983 17329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "286f8b8e33c4427e886d8b0765bfce65"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "91ec4736df952b00358fc41daa08745c"
server_key_iv: "f215bc9ddd8ab030422916931bca7f08"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.043534 17329 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:01:19.045862 17359 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.047060 17329 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:01:19.047187 17329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "286f8b8e33c4427e886d8b0765bfce65"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "91ec4736df952b00358fc41daa08745c"
server_key_iv: "f215bc9ddd8ab030422916931bca7f08"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.047293 17329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:19.063484 17329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:19.066296 17329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:19.066502 17329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:19.067162 17329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:19.068007 17329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:19.068080 17329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.068156 17329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:19.068199 17329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.077301 17329 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:44829
I20260810 19:01:19.077344 17542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:44829 every 8 connection(s)
I20260810 19:01:19.078159 17329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:01:19.083142 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 17329
I20260810 19:01:19.083298 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:01:19.083652 28238 external_mini_cluster.cc:1428] Setting key bbc66d1cf5bf012a1fa5ee3780225e76
Aug 10 19:01:19 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:19.092180 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.079921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:54517 (local address 127.27.147.190:41613)
0810 19:01:19.080064 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.080068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.080849 (+ 781us) server_negotiation.cc:407] Connection header received
0810 19:01:19.082131 (+ 1282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.082134 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.082181 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.082255 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:19.083568 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.084277 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.085513 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.085745 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.085860 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.088606 (+ 2746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:19.088615 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:19.088617 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:19.088640 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:19.090133 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.090652 (+ 519us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.090663 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.090665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.090742 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.091111 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.091116 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.091118 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.091295 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:19.091405 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.091883 (+ 478us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.092007 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:01:19.092916 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.080260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:54517)
0810 19:01:19.080698 (+ 438us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.080740 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.081930 (+ 1190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.082416 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.082423 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.082772 (+ 349us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:19.083393 (+ 621us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.083416 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.084406 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.084411 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.085381 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.085391 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.085901 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.085905 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.086067 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.086638 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:19.086654 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:19.088441 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:19.090266 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:19.090272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:19.090285 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:19.090541 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:19.090860 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:19.090863 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:19.090865 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:19.091011 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:19.091415 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:19.091423 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:19.091647 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.091824 (+ 177us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.092135 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":123,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 19:01:19.094292 17543 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41613
I20260810 19:01:19.094527 17543 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:19.095126 17543 heartbeater.cc:511] Master 127.27.147.190:41613 requested a full tablet report, sending...
I20260810 19:01:19.096675 17221 ts_manager.cc:194] Registered new tserver with Master: 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129:44829)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 19:01:19.097896 17221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:54517
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 19:01:19.112200 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41613
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:19.223620 17554 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:19.223851 17554 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:19.223892 17554 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:19.227805 17554 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:19.227866 17554 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:19.227922 17554 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:01:19.232879 17554 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17554
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:19.233747 17554 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:19.234637 17554 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:19.244421 17560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:19.244427 17561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:19.244526 17554 server_base.cc:1034] running on GCE node
W20260810 19:01:19.244439 17563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:19.245139 17554 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:19.245739 17554 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:19.246901 17554 hybrid_clock.cc:648] HybridClock initialized: now 1786388479246876 us; error 26 us; skew 500 ppm
Aug 10 19:01:19 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:19.249679 17554 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:01:19.250626 17554 webserver.cc:460] Webserver started at http://127.27.147.130:33525/ using document root <none> and password file <none>
I20260810 19:01:19.251334 17554 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:19.251425 17554 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:19.251633 17554 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:19.253621 17554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "13769f01eaff4755b2509c0ff701062c"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "9afa5a0b7607f191671593afbd858cbf"
server_key_iv: "92a3c62f256a9c008c5e14afb14c5635"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.254194 17554 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "13769f01eaff4755b2509c0ff701062c"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "9afa5a0b7607f191671593afbd858cbf"
server_key_iv: "92a3c62f256a9c008c5e14afb14c5635"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.258495 17554 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.007s sys 0.000s
I20260810 19:01:19.260865 17577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.262007 17554 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:19.262142 17554 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "13769f01eaff4755b2509c0ff701062c"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "9afa5a0b7607f191671593afbd858cbf"
server_key_iv: "92a3c62f256a9c008c5e14afb14c5635"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.262250 17554 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:19.278519 17554 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:19.281812 17554 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:19.282025 17554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:19.282614 17554 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:19.283486 17554 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:19.283562 17554 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.283632 17554 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:19.283680 17554 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.292730 17554 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:35673
I20260810 19:01:19.292766 17779 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:35673 every 8 connection(s)
I20260810 19:01:19.293599 17554 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:01:19.298988 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 17554
I20260810 19:01:19.299139 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:01:19.299476 28238 external_mini_cluster.cc:1428] Setting key b0d070215c2ddbbb4d3fb98597afa695
Aug 10 19:01:19 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:19.307691 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.295375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:34917 (local address 127.27.147.190:41613)
0810 19:01:19.295545 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.295550 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.296364 (+ 814us) server_negotiation.cc:407] Connection header received
0810 19:01:19.297678 (+ 1314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.297681 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.297727 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.297811 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:19.299011 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.299840 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.301148 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.301512 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.301633 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.304081 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:19.304084 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:19.304086 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:19.304108 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:19.305559 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.306087 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.306092 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.306094 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.306178 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.306522 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.306527 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.306529 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.306802 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:19.306925 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.307396 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.307521 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 19:01:19.308480 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.295771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:34917)
0810 19:01:19.296208 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.296251 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.297487 (+ 1236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.297974 (+ 487us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.297980 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.298309 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:19.298788 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.298801 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.299983 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.299987 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.301030 (+ 1043us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.301039 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.301655 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.301658 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.301790 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.302378 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:19.302394 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:19.303914 (+ 1520us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:19.305695 (+ 1781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:19.305700 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:19.305716 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:19.305980 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:19.306274 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:19.306286 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:19.306288 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:19.306425 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:19.306923 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:19.306930 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:19.307157 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.307341 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.307639 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":119,"threads_started":1}
I20260810 19:01:19.309653 17782 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41613
I20260810 19:01:19.309927 17782 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:19.310420 17782 heartbeater.cc:511] Master 127.27.147.190:41613 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 19:01:19.311591 17221 ts_manager.cc:194] Registered new tserver with Master: 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
I20260810 19:01:19.312232 17221 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:34917
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 19:01:19.329396 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41613
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:19.443490 17791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:19.443718 17791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:19.443759 17791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:19.447742 17791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:19.447794 17791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:19.447849 17791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:01:19.453043 17791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:19.453969 17791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:19.454991 17791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:19.464427 17812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:19.464437 17806 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:19.464438 17805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:19.465150 17791 server_base.cc:1034] running on GCE node
I20260810 19:01:19.465656 17791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:19.466378 17791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:19.467571 17791 hybrid_clock.cc:648] HybridClock initialized: now 1786388479467555 us; error 31 us; skew 500 ppm
Aug 10 19:01:19 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:19.470242 17791 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:01:19.471210 17791 webserver.cc:460] Webserver started at http://127.27.147.131:34331/ using document root <none> and password file <none>
I20260810 19:01:19.471763 17791 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:19.471858 17791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:19.472090 17791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:19.474028 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "f7bae4f49a064686b780dccd8e364c70"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "5e182be7ed83d6757292406433d4897d"
server_key_iv: "cb95d280e6b4059b5810120d8e62bd94"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.474596 17791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "f7bae4f49a064686b780dccd8e364c70"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "5e182be7ed83d6757292406433d4897d"
server_key_iv: "cb95d280e6b4059b5810120d8e62bd94"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.478212 17791 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:01:19.480619 17820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.481796 17791 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 19:01:19.481941 17791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "f7bae4f49a064686b780dccd8e364c70"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "5e182be7ed83d6757292406433d4897d"
server_key_iv: "cb95d280e6b4059b5810120d8e62bd94"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.482051 17791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:19.508322 17791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:19.511760 17791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:19.511981 17791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:19.512593 17791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:19.513438 17791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:19.513515 17791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.513581 17791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:19.513634 17791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.522779 17791 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:33889
I20260810 19:01:19.522886 18008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:33889 every 8 connection(s)
I20260810 19:01:19.523720 17791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:01:19.526021 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 17791
I20260810 19:01:19.526165 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:01:19.526525 28238 external_mini_cluster.cc:1428] Setting key 743201cdc7a9fc5f58b86a4e19fea357
Aug 10 19:01:19 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:19.537953 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.525553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:32813 (local address 127.27.147.190:41613)
0810 19:01:19.525722 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.525726 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.526656 (+ 930us) server_negotiation.cc:407] Connection header received
0810 19:01:19.528554 (+ 1898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.528558 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.528612 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.528692 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:19.529817 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.530365 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.531530 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.531753 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.531884 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.534444 (+ 2560us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:19.534447 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:19.534450 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:19.534482 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:19.535924 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.536448 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.536451 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.536452 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.536524 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.536873 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.536878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.536879 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.537057 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:19.537164 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.537644 (+ 480us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.537766 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:01:19.538873 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.525939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:32813)
0810 19:01:19.526492 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.526534 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.528347 (+ 1813us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.528862 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.528869 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.529208 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:19.529670 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.529683 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.530491 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.530495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.531395 (+ 900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.531405 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.531923 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.531927 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.532093 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.532716 (+ 623us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:19.532742 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:19.534222 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:19.536046 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:19.536052 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:19.536066 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:19.536326 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:19.536619 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:19.536623 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:19.536625 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:19.536769 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:19.537164 (+ 395us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:19.537172 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:19.537414 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.537599 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.537912 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":161,"threads_started":1}
I20260810 19:01:19.540030 18010 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41613
I20260810 19:01:19.540314 18010 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:19.540822 18010 heartbeater.cc:511] Master 127.27.147.190:41613 requested a full tablet report, sending...
I20260810 19:01:19.541810 17220 ts_manager.cc:194] Registered new tserver with Master: f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889)
I20260810 19:01:19.542419 17220 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:32813
I20260810 19:01:19.552312 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:01:19.564620 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.554288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.27.147.190:41613)
0810 19:01:19.554434 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.554441 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.554629 (+ 188us) server_negotiation.cc:407] Connection header received
0810 19:01:19.554780 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.554784 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.554863 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.554956 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:19.555942 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.556762 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.558230 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.558657 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.558780 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.560429 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:19.560433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:19.560436 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:19.560472 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:19.562365 (+ 1893us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.562847 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.562853 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.562865 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.562943 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:19.563212 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:19.563217 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:19.563221 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:19.563521 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:19.563640 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.564263 (+ 623us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.564398 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:01:19.569384 17220 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60652:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:01:19.572453 17220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:01:19.590001 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.581122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42808 (local address 127.27.147.130:35673)
0810 19:01:19.584497 (+ 3375us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.584513 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.584539 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:01:19.584753 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.584758 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.584948 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.585105 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:19.585601 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.586726 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.588277 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.589125 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.589257 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.589426 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.589729 (+ 303us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.589833 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3060,"thread_start_us":2182,"threads_started":1}
I20260810 19:01:19.590034 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.581687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44829 (local address 127.0.0.1:50680)
0810 19:01:19.583570 (+ 1883us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.583618 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.583756 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.585049 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.585055 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.585080 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:19.585218 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.585228 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.586697 (+ 1469us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.586700 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.588474 (+ 1774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.588482 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.589311 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.589314 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.589481 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.589566 (+ 85us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.589674 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.589849 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1835,"mutex_wait_us":527,"spinlock_wait_cycles":4224,"thread_start_us":1616,"threads_started":1}
I20260810 19:01:19.590034 18033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.580996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:35673 (local address 127.0.0.1:42808)
0810 19:01:19.583567 (+ 2571us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.583618 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.584578 (+ 960us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.585270 (+ 692us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.585273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.585296 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:19.585487 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.585495 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.586874 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.586878 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.588148 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.588160 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.589273 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.589280 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.589488 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.589566 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.589681 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.589849 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2388,"mutex_wait_us":176,"spinlock_wait_cycles":8704,"thread_start_us":74,"threads_started":1}
I20260810 19:01:19.590864 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.583587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:33889 (local address 127.0.0.1:46344)
0810 19:01:19.583838 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.583852 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.583961 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.585270 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.585273 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.585301 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:19.585497 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.585503 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.588027 (+ 2524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.588032 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.589219 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.589229 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.590413 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.590419 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.590518 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.590546 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.590614 (+ 68us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.590689 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":59,"spinlock_wait_cycles":5632,"thread_start_us":99,"threads_started":1}
I20260810 19:01:19.590931 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.584077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50680 (local address 127.27.147.129:44829)
0810 19:01:19.584553 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.584569 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.584587 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:01:19.584669 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.584677 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.584869 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.585032 (+ 163us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:19.585347 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.586584 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.588595 (+ 2011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.589159 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.590474 (+ 1315us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.590614 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.590666 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.590759 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":131,"threads_started":1}
I20260810 19:01:19.591387 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.584049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.27.147.131:33889)
0810 19:01:19.584620 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.584629 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.584673 (+ 44us) server_negotiation.cc:407] Connection header received
0810 19:01:19.584750 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.584755 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.584948 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.585105 (+ 157us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:19.586936 (+ 1831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.587892 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.589351 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.590088 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.590208 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.590343 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.591134 (+ 791us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.591251 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":177,"threads_started":1}
I20260810 19:01:19.593456 17665 tablet_service.cc:1467] Processing CreateTablet for tablet bd1bdfb69d6b421a95fa08c6bf93672e (DEFAULT_TABLE table=test-table [id=5edc93fab27944f7a4b5ea903d67770a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:19.593622 17426 tablet_service.cc:1467] Processing CreateTablet for tablet bd1bdfb69d6b421a95fa08c6bf93672e (DEFAULT_TABLE table=test-table [id=5edc93fab27944f7a4b5ea903d67770a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:19.593835 17901 tablet_service.cc:1467] Processing CreateTablet for tablet bd1bdfb69d6b421a95fa08c6bf93672e (DEFAULT_TABLE table=test-table [id=5edc93fab27944f7a4b5ea903d67770a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:19.594398 17665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1bdfb69d6b421a95fa08c6bf93672e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:19.594666 17901 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1bdfb69d6b421a95fa08c6bf93672e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:19.594661 17426 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd1bdfb69d6b421a95fa08c6bf93672e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:19.600600 18040 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Bootstrap starting.
I20260810 19:01:19.602404 18041 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap starting.
I20260810 19:01:19.603178 18045 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Bootstrap starting.
I20260810 19:01:19.603543 18040 tablet_bootstrap.cc:654] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:19.604269 18040 log.cc:824] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:19.604683 18041 tablet_bootstrap.cc:654] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:19.605386 18045 tablet_bootstrap.cc:654] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:19.605648 18041 log.cc:824] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:19.606698 18040 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: No bootstrap required, opened a new log
I20260810 19:01:19.606736 18045 log.cc:824] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:19.606945 18040 ts_tablet_manager.cc:1397] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.004s
I20260810 19:01:19.607935 18041 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: No bootstrap required, opened a new log
I20260810 19:01:19.608178 18041 ts_tablet_manager.cc:1397] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:01:19.608700 18045 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: No bootstrap required, opened a new log
I20260810 19:01:19.608883 18045 ts_tablet_manager.cc:1397] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260810 19:01:19.610373 18040 raft_consensus.cc:348] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.610622 18040 raft_consensus.cc:374] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:19.610679 18040 raft_consensus.cc:729] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13769f01eaff4755b2509c0ff701062c, State: Initialized, Role: FOLLOWER
I20260810 19:01:19.611162 18040 consensus_queue.cc:260] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.612205 18040 ts_tablet_manager.cc:1428] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:01:19.612640 17782 heartbeater.cc:503] Master 127.27.147.190:41613 was elected leader, sending a full tablet report...
I20260810 19:01:19.612890 18045 raft_consensus.cc:348] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.612890 18041 raft_consensus.cc:348] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.613126 18045 raft_consensus.cc:374] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:19.613126 18041 raft_consensus.cc:374] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:19.613178 18041 raft_consensus.cc:729] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286f8b8e33c4427e886d8b0765bfce65, State: Initialized, Role: FOLLOWER
I20260810 19:01:19.613178 18045 raft_consensus.cc:729] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7bae4f49a064686b780dccd8e364c70, State: Initialized, Role: FOLLOWER
I20260810 19:01:19.613591 18045 consensus_queue.cc:260] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.613734 18041 consensus_queue.cc:260] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.614249 18010 heartbeater.cc:503] Master 127.27.147.190:41613 was elected leader, sending a full tablet report...
I20260810 19:01:19.614543 18045 ts_tablet_manager.cc:1428] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260810 19:01:19.614670 17543 heartbeater.cc:503] Master 127.27.147.190:41613 was elected leader, sending a full tablet report...
I20260810 19:01:19.615445 18041 ts_tablet_manager.cc:1428] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260810 19:01:19.683954 18055 raft_consensus.cc:482] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:19.684161 18055 raft_consensus.cc:504] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.685494 18055 leader_election.cc:288] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673), 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129:44829)
I20260810 19:01:19.689750 18031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.685906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:38157 (local address 127.27.147.130:35673)
0810 19:01:19.686046 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.686067 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.686087 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:01:19.686150 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.686155 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.686209 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.686322 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:19.686725 (+ 403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.687413 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.688607 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.689161 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.689254 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.689298 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.689534 (+ 236us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.689616 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:01:19.689755 18012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.685734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:35673 (local address 127.27.147.131:38157)
0810 19:01:19.685871 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.685889 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.686007 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.686396 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.686399 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.686422 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:19.686565 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.686573 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.687545 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.687549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.688470 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.688480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.689283 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.689287 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.689393 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.689422 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.689511 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.689595 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 19:01:19.690168 18037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.686242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:43491 (local address 127.27.147.129:44829)
0810 19:01:19.686383 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.686387 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.686573 (+ 186us) server_negotiation.cc:407] Connection header received
0810 19:01:19.686662 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.686665 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.686726 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.686786 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:19.687135 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.687789 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.688963 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.689541 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.689663 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.689712 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.689935 (+ 223us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.690052 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:01:19.690526 18061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.686056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44829 (local address 127.27.147.131:43491)
0810 19:01:19.686472 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.686484 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:19.686574 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:19.686843 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:19.686846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:19.686864 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:19.686983 (+ 119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.686990 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.687922 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.687925 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.688817 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:19.688833 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.689662 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.689665 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:19.689765 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.689797 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:01:19.690325 (+ 528us) client_negotiation.cc:240] Negotiation successful
0810 19:01:19.690420 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":318,"mutex_wait_us":2,"thread_start_us":143,"threads_started":1}
I20260810 19:01:19.690690 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1bdfb69d6b421a95fa08c6bf93672e" candidate_uuid: "f7bae4f49a064686b780dccd8e364c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13769f01eaff4755b2509c0ff701062c" is_pre_election: true
I20260810 19:01:19.691097 17697 raft_consensus.cc:2454] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7bae4f49a064686b780dccd8e364c70 in term 0.
I20260810 19:01:19.691193 17463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1bdfb69d6b421a95fa08c6bf93672e" candidate_uuid: "f7bae4f49a064686b780dccd8e364c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286f8b8e33c4427e886d8b0765bfce65" is_pre_election: true
I20260810 19:01:19.691514 17463 raft_consensus.cc:2454] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f7bae4f49a064686b780dccd8e364c70 in term 0.
I20260810 19:01:19.691645 17825 leader_election.cc:302] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13769f01eaff4755b2509c0ff701062c, f7bae4f49a064686b780dccd8e364c70; no voters:
I20260810 19:01:19.691963 18055 raft_consensus.cc:2781] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:01:19.692059 18055 raft_consensus.cc:482] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:01:19.692106 18055 raft_consensus.cc:3037] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:19.693184 18055 raft_consensus.cc:504] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.693524 18055 leader_election.cc:288] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [CANDIDATE]: Term 1 election: Requested vote from peers 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673), 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129:44829)
I20260810 19:01:19.693944 17697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1bdfb69d6b421a95fa08c6bf93672e" candidate_uuid: "f7bae4f49a064686b780dccd8e364c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13769f01eaff4755b2509c0ff701062c"
I20260810 19:01:19.693933 17463 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd1bdfb69d6b421a95fa08c6bf93672e" candidate_uuid: "f7bae4f49a064686b780dccd8e364c70" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "286f8b8e33c4427e886d8b0765bfce65"
I20260810 19:01:19.694070 17697 raft_consensus.cc:3037] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:19.694072 17463 raft_consensus.cc:3037] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:19.695453 17697 raft_consensus.cc:2454] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7bae4f49a064686b780dccd8e364c70 in term 1.
I20260810 19:01:19.695453 17463 raft_consensus.cc:2454] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f7bae4f49a064686b780dccd8e364c70 in term 1.
I20260810 19:01:19.695729 17825 leader_election.cc:302] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13769f01eaff4755b2509c0ff701062c, f7bae4f49a064686b780dccd8e364c70; no voters:
I20260810 19:01:19.695919 18055 raft_consensus.cc:2781] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:19.696146 18055 raft_consensus.cc:686] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 1 LEADER]: Becoming Leader. State: Replica: f7bae4f49a064686b780dccd8e364c70, State: Running, Role: LEADER
I20260810 19:01:19.696501 18055 consensus_queue.cc:237] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:19.699934 17221 catalog_manager.cc:5495] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 reported cstate change: term changed from 0 to 1, leader changed from <none> to f7bae4f49a064686b780dccd8e364c70 (127.27.147.131). New cstate: current_term: 1 leader_uuid: "f7bae4f49a064686b780dccd8e364c70" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } health_report { overall_health: HEALTHY } } }
I20260810 19:01:19.710747 18038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.707425 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46360 (local address 127.27.147.131:33889)
0810 19:01:19.707546 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:19.707553 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:19.707663 (+ 110us) server_negotiation.cc:407] Connection header received
0810 19:01:19.707764 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:19.707766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:19.707848 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:19.707955 (+ 107us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:01:19.708174 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.708661 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.709449 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:19.709649 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:19.709724 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:19.709979 (+ 255us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:01:19.710363 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:01:19.710456 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:19.710514 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 19:01:19.710560 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:01:19.716542 17697 raft_consensus.cc:1261] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Refusing update from remote peer f7bae4f49a064686b780dccd8e364c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:19.716542 17463 raft_consensus.cc:1261] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Refusing update from remote peer f7bae4f49a064686b780dccd8e364c70: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:19.717314 18055 consensus_queue.cc:1035] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:19.717501 18062 consensus_queue.cc:1035] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:19.727659 18076 mvcc.cc:205] Tried to move back new op lower bound from 7317047212911239168 to 7317047212840226816. Current Snapshot: MvccSnapshot[applied={T|T < 7317047212911239168}]
I20260810 19:01:19.734431 17221 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:19.734581 17221 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60652:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:19.737056 17221 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Sending DeleteTablet for 3 replicas of tablet bd1bdfb69d6b421a95fa08c6bf93672e
I20260810 19:01:19.737885 17901 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1bdfb69d6b421a95fa08c6bf93672e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:19 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:46344
I20260810 19:01:19.737888 17665 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1bdfb69d6b421a95fa08c6bf93672e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:19 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:42808
I20260810 19:01:19.738181 17426 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1bdfb69d6b421a95fa08c6bf93672e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:19 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:50680
I20260810 19:01:19.738458 18092 tablet_replica.cc:330] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: stopping tablet replica
I20260810 19:01:19.738686 18093 tablet_replica.cc:330] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: stopping tablet replica
I20260810 19:01:19.738714 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 17329
I20260810 19:01:19.751633 18092 raft_consensus.cc:2229] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:19.752321 18092 raft_consensus.cc:2258] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:19.754473 18091 tablet_replica.cc:330] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: stopping tablet replica
I20260810 19:01:19.755203 18091 raft_consensus.cc:2229] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:01:19.755700 18091 raft_consensus.cc:2258] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:19.756143 18092 ts_tablet_manager.cc:1905] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:19.758383 18091 ts_tablet_manager.cc:1905] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:19.759441 18092 ts_tablet_manager.cc:1918] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:19.759619 18092 log.cc:1196] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/bd1bdfb69d6b421a95fa08c6bf93672e
I20260810 19:01:19.760085 18092 ts_tablet_manager.cc:1939] T bd1bdfb69d6b421a95fa08c6bf93672e P 13769f01eaff4755b2509c0ff701062c: Deleting consensus metadata
I20260810 19:01:19.761183 18091 ts_tablet_manager.cc:1918] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:19.761286 18091 log.cc:1196] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/bd1bdfb69d6b421a95fa08c6bf93672e
I20260810 19:01:19.761659 18091 ts_tablet_manager.cc:1939] T bd1bdfb69d6b421a95fa08c6bf93672e P f7bae4f49a064686b780dccd8e364c70: Deleting consensus metadata
I20260810 19:01:19.761708 17197 catalog_manager.cc:4841] TS 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673): tablet bd1bdfb69d6b421a95fa08c6bf93672e (table test-table [id=5edc93fab27944f7a4b5ea903d67770a]) successfully deleted
I20260810 19:01:19.762688 17196 catalog_manager.cc:4841] TS f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889): tablet bd1bdfb69d6b421a95fa08c6bf93672e (table test-table [id=5edc93fab27944f7a4b5ea903d67770a]) successfully deleted
W20260810 19:01:19.763441 17197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:44829 (error 108)
I20260810 19:01:19.763681 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 17554
I20260810 19:01:19.771104 18036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:19.770650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44829 (local address 127.0.0.1:50690)
0810 19:01:19.770920 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:19.771003 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
W20260810 19:01:19.771325 17197 catalog_manager.cc:4584] TS 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129:44829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bd1bdfb69d6b421a95fa08c6bf93672e: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44829: connect: Connection refused (error 111)
I20260810 19:01:19.785109 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 17791
I20260810 19:01:19.796097 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 17172
I20260810 19:01:19.807135 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41613
--webserver_interface=127.27.147.190
--webserver_port=45505
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:19.917492 18095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:19.917752 18095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:19.917848 18095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:19.921824 18095 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:19.921903 18095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:19.921943 18095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:19.922003 18095 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:19.922043 18095 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:19.927119 18095 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:19.928067 18095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:19.929013 18095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:19.943692 18106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:19.943753 18105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:19.943708 18109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:19.944072 18095 server_base.cc:1034] running on GCE node
I20260810 19:01:19.944645 18095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:19.945721 18095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:19.946883 18095 hybrid_clock.cc:648] HybridClock initialized: now 1786388479946869 us; error 29 us; skew 500 ppm
Aug 10 19:01:19 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:19.949549 18095 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:19.950418 18095 webserver.cc:460] Webserver started at http://127.27.147.190:45505/ using document root <none> and password file <none>
I20260810 19:01:19.951061 18095 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:19.951181 18095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:19.955152 18095 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:01:19.957219 18120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:19.958369 18095 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:19.958528 18095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "56773a037d81472484f5dc475a2742f8"
format_stamp: "Formatted at 2026-08-10 19:01:18 on dist-test-slave-znh6"
server_key: "b447cfcdbd530d55a2ddd24d9988c66d"
server_key_iv: "78f02d729a73e0d582716ce436d17c0c"
server_key_version: "encryptionkey@0"
I20260810 19:01:19.959158 18095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:19.974977 18095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:19.977598 18095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:19.977785 18095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:19.984752 18095 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:41613
I20260810 19:01:19.985033 18217 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:41613 every 8 connection(s)
I20260810 19:01:19.985826 18095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:19.990130 18218 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:01:19.991797 18218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Bootstrap starting.
I20260810 19:01:19.993508 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 18095
I20260810 19:01:19.993985 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:44829
--local_ip_for_outbound_sockets=127.27.147.129
--tserver_master_addrs=127.27.147.190:41613
--webserver_port=46327
--webserver_interface=127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:20.009929 18218 log.cc:824] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:20.019723 18218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:01:20.020103 18218 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Bootstrap complete.
I20260810 19:01:20.024331 18218 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:20.024739 18218 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56773a037d81472484f5dc475a2742f8, State: Initialized, Role: FOLLOWER
I20260810 19:01:20.025231 18218 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:20.025408 18218 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:20.025496 18218 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:20.025620 18218 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:01:20.027576 18218 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:20.027920 18218 leader_election.cc:302] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56773a037d81472484f5dc475a2742f8; no voters:
I20260810 19:01:20.028268 18218 leader_election.cc:288] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:01:20.028345 18224 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:01:20.028556 18224 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 LEADER]: Becoming Leader. State: Replica: 56773a037d81472484f5dc475a2742f8, State: Running, Role: LEADER
I20260810 19:01:20.028857 18224 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:20.029533 18218 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:20.029799 18224 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56773a037d81472484f5dc475a2742f8. Latest consensus state: current_term: 2 leader_uuid: "56773a037d81472484f5dc475a2742f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } } }
I20260810 19:01:20.029914 18224 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:20.030514 18225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "56773a037d81472484f5dc475a2742f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } } }
I20260810 19:01:20.030628 18225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:20.030833 18230 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:20.035665 18230 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:20.036309 18230 catalog_manager.cc:1259] Loaded cluster ID: 8d79316e09024497982c62942722d8ee
I20260810 19:01:20.036374 18230 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:20.037815 18230 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 19:01:20.042943 18230 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 19:01:20.048221 18230 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:20.048939 18230 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Loaded TSK: 0
I20260810 19:01:20.050063 18230 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:01:20.113123 18222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:20.113379 18222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:20.113456 18222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:20.117408 18222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:20.117486 18222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:20.117570 18222 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:01:20.122639 18222 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:44829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=46327
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:20.123612 18222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:20.124554 18222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:20.135481 18264 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.135564 18265 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.135548 18268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:20.135684 18222 server_base.cc:1034] running on GCE node
I20260810 19:01:20.136224 18222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:20.136873 18222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:20.138060 18222 hybrid_clock.cc:648] HybridClock initialized: now 1786388480138031 us; error 42 us; skew 500 ppm
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:20.141292 18222 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:01:20.142290 18222 webserver.cc:460] Webserver started at http://127.27.147.129:46327/ using document root <none> and password file <none>
I20260810 19:01:20.142908 18222 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:20.143013 18222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:20.146909 18222 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:01:20.149053 18275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.150331 18222 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:20.150427 18222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "286f8b8e33c4427e886d8b0765bfce65"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "91ec4736df952b00358fc41daa08745c"
server_key_iv: "f215bc9ddd8ab030422916931bca7f08"
server_key_version: "encryptionkey@0"
I20260810 19:01:20.151084 18222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:20.161253 18222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:20.164517 18222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:20.164719 18222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:20.165341 18222 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:20.166424 18286 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 19:01:20.170509 18222 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 19:01:20.170573 18222 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 19:01:20.170639 18222 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 19:01:20.172320 18222 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 19:01:20.172379 18222 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 19:01:20.172636 18286 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap starting.
I20260810 19:01:20.183421 18222 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:44829
I20260810 19:01:20.183430 18479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:44829 every 8 connection(s)
I20260810 19:01:20.184578 18222 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:01:20.191375 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 18222
I20260810 19:01:20.192032 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:35673
--local_ip_for_outbound_sockets=127.27.147.130
--tserver_master_addrs=127.27.147.190:41613
--webserver_port=33525
--webserver_interface=127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:20.194401 18286 log.cc:824] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:20.202118 18286 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:01:20.202562 18286 tablet_bootstrap.cc:492] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap complete.
I20260810 19:01:20.203125 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.186874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:52941 (local address 127.27.147.190:41613)
0810 19:01:20.187294 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.187309 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.187718 (+ 409us) server_negotiation.cc:407] Connection header received
0810 19:01:20.188808 (+ 1090us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.188819 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.189094 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.189429 (+ 335us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:20.190551 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.191695 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.193020 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.193480 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.193662 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.196704 (+ 3042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:20.196709 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:20.196721 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:20.196756 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:20.199109 (+ 2353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.199828 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.199834 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.199839 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.200021 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.200491 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.200495 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.200497 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.200947 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:20.201084 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.201814 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.202132 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":292,"thread_start_us":108,"threads_started":1}
I20260810 19:01:20.203410 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.186773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:52941)
0810 19:01:20.187548 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.187587 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.188398 (+ 811us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.189489 (+ 1091us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.189497 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.189918 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:20.190367 (+ 449us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.190380 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.191874 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.191877 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.192885 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.192896 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.193632 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.193636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.193841 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.194581 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:20.194607 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:20.196493 (+ 1886us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:20.199271 (+ 2778us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.199280 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.199298 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.199689 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.200146 (+ 457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.200152 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.200158 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.200368 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.201087 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:20.201096 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:20.201438 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.201671 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.202055 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":584,"thread_start_us":167,"threads_started":1}
I20260810 19:01:20.204084 18286 ts_tablet_manager.cc:1397] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.009s
I20260810 19:01:20.206131 18480 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41613
I20260810 19:01:20.206406 18480 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:20.207036 18480 heartbeater.cc:511] Master 127.27.147.190:41613 requested a full tablet report, sending...
I20260810 19:01:20.206766 18286 raft_consensus.cc:348] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:20.207263 18286 raft_consensus.cc:729] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286f8b8e33c4427e886d8b0765bfce65, State: Initialized, Role: FOLLOWER
I20260810 19:01:20.207821 18286 consensus_queue.cc:260] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } } peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } }
I20260810 19:01:20.208802 18286 ts_tablet_manager.cc:1428] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:01:20.210347 18151 ts_manager.cc:194] Registered new tserver with Master: 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129:44829)
I20260810 19:01:20.213603 18151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:52941
I20260810 19:01:20.216066 18480 heartbeater.cc:503] Master 127.27.147.190:41613 was elected leader, sending a full tablet report...
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388479, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 19:01:20.223845 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.213046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44829 (local address 127.0.0.1:50702)
0810 19:01:20.213748 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.213789 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.213916 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.214485 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.214492 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.214984 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:20.215108 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.215116 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.216579 (+ 1463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.216583 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.217581 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.217594 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.218141 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.218145 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.218266 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.218781 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:20.218795 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:20.220312 (+ 1517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:20.222068 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.222079 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.222084 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.222307 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.222625 (+ 318us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.222629 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.222631 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.222720 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.223198 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:20.223203 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:20.223341 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.223488 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.223672 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":109,"threads_started":1}
I20260810 19:01:20.223865 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.213138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50702 (local address 127.27.147.129:44829)
0810 19:01:20.213504 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.213516 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.214049 (+ 533us) server_negotiation.cc:407] Connection header received
0810 19:01:20.214116 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.214121 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.214316 (+ 195us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.214485 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:20.215241 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.216433 (+ 1192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.217702 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.218023 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.218109 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.220464 (+ 2355us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:20.220473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:20.220482 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:20.220503 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:20.221953 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.222447 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.222452 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.222453 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.222529 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.222839 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.222845 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.222850 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.223077 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:20.223221 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.223570 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.223725 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":283,"thread_start_us":104,"threads_started":1}
I20260810 19:01:20.224776 18359 tablet_service.cc:1514] Processing DeleteTablet for tablet bd1bdfb69d6b421a95fa08c6bf93672e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:19 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:50702
I20260810 19:01:20.225684 18498 tablet_replica.cc:330] stopping tablet replica
I20260810 19:01:20.225978 18498 raft_consensus.cc:2229] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:20.226210 18498 raft_consensus.cc:2258] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:20.227144 18498 ts_tablet_manager.cc:1905] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:20.230655 18498 ts_tablet_manager.cc:1918] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:20.230767 18498 log.cc:1196] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/bd1bdfb69d6b421a95fa08c6bf93672e
I20260810 19:01:20.231081 18498 ts_tablet_manager.cc:1939] T bd1bdfb69d6b421a95fa08c6bf93672e P 286f8b8e33c4427e886d8b0765bfce65: Deleting consensus metadata
I20260810 19:01:20.232686 18126 catalog_manager.cc:4841] TS 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129:44829): tablet bd1bdfb69d6b421a95fa08c6bf93672e (table test-table [id=5edc93fab27944f7a4b5ea903d67770a]) successfully deleted
W20260810 19:01:20.316586 18492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:20.316828 18492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:20.316869 18492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:20.320991 18492 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:20.321053 18492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:20.321146 18492 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:01:20.326561 18492 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:35673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=33525
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18492
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:20.327682 18492 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:20.328747 18492 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:20.339075 18508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.339078 18510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.339087 18514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:20.339372 18492 server_base.cc:1034] running on GCE node
I20260810 19:01:20.339951 18492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:20.340683 18492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:20.341897 18492 hybrid_clock.cc:648] HybridClock initialized: now 1786388480341890 us; error 36 us; skew 500 ppm
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:20.344826 18492 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:01:20.345630 18492 webserver.cc:460] Webserver started at http://127.27.147.130:33525/ using document root <none> and password file <none>
I20260810 19:01:20.346159 18492 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:20.346233 18492 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:20.350135 18492 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:20.352464 18533 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.353848 18492 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:01:20.353991 18492 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "13769f01eaff4755b2509c0ff701062c"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "9afa5a0b7607f191671593afbd858cbf"
server_key_iv: "92a3c62f256a9c008c5e14afb14c5635"
server_key_version: "encryptionkey@0"
I20260810 19:01:20.354586 18492 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:20.380412 18492 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:20.383471 18492 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:20.383694 18492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:20.384313 18492 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:20.385159 18492 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:20.385207 18492 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.385277 18492 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:20.385319 18492 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.394608 18492 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:35673
I20260810 19:01:20.394630 18719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:35673 every 8 connection(s)
I20260810 19:01:20.395691 18492 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:20.409061 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 18492
I20260810 19:01:20.409562 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:33889
--local_ip_for_outbound_sockets=127.27.147.131
--tserver_master_addrs=127.27.147.190:41613
--webserver_port=34331
--webserver_interface=127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:20.415922 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.397521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:33211 (local address 127.27.147.190:41613)
0810 19:01:20.397670 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.397675 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.398617 (+ 942us) server_negotiation.cc:407] Connection header received
0810 19:01:20.400693 (+ 2076us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.400697 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.400771 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.400893 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:20.402500 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.403492 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.405145 (+ 1653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.405488 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.405558 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.408768 (+ 3210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:20.408774 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:20.408777 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:20.408805 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:20.413706 (+ 4901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.414234 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.414238 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.414240 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.414331 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.414679 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.414684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.414686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.414964 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:20.415076 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.415587 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.415731 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:01:20.416759 18723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.397878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:33211)
0810 19:01:20.398423 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.398483 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.400418 (+ 1935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.401108 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.401119 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.401583 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:20.402278 (+ 695us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.402320 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.403675 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.403678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.405007 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.405018 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.405754 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.405757 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.405953 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.406637 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:20.406661 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:20.408572 (+ 1911us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:20.413828 (+ 5256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.413835 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.413851 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.414121 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.414431 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.414435 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.414437 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.414577 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.415075 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:20.415082 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:20.415338 (+ 256us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.415545 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.415885 (+ 340us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308,"thread_start_us":123,"threads_started":1}
I20260810 19:01:20.418089 18721 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41613
I20260810 19:01:20.418354 18721 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:20.418953 18721 heartbeater.cc:511] Master 127.27.147.190:41613 requested a full tablet report, sending...
I20260810 19:01:20.420114 18151 ts_manager.cc:194] Registered new tserver with Master: 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
I20260810 19:01:20.420701 18151 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:33211
W20260810 19:01:20.525029 18724 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:20.525281 18724 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:20.525360 18724 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:20.529414 18724 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:20.529484 18724 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:20.529564 18724 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:01:20.534762 18724 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:33889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=34331
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18724
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:20.535751 18724 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:20.536689 18724 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:20.546434 18730 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.546434 18731 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.546411 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:20.546736 18724 server_base.cc:1034] running on GCE node
I20260810 19:01:20.547283 18724 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:20.547902 18724 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:20.549083 18724 hybrid_clock.cc:648] HybridClock initialized: now 1786388480549066 us; error 35 us; skew 500 ppm
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:20.551923 18724 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:01:20.552830 18724 webserver.cc:460] Webserver started at http://127.27.147.131:34331/ using document root <none> and password file <none>
I20260810 19:01:20.553367 18724 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:20.553470 18724 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:20.557353 18724 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:01:20.559425 18741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.560645 18724 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:20.560794 18724 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "f7bae4f49a064686b780dccd8e364c70"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "5e182be7ed83d6757292406433d4897d"
server_key_iv: "cb95d280e6b4059b5810120d8e62bd94"
server_key_version: "encryptionkey@0"
I20260810 19:01:20.561309 18724 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:20.587749 18724 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:20.591058 18724 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:20.591257 18724 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:20.591856 18724 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:20.592674 18724 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:20.592746 18724 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.592816 18724 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:20.592873 18724 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.602139 18724 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:33889
I20260810 19:01:20.602177 18941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:33889 every 8 connection(s)
I20260810 19:01:20.603173 18724 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:01:20.608100 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 18724
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:20.617455 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.608780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60660 (local address 127.27.147.190:41613)
0810 19:01:20.609042 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.609049 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.609210 (+ 161us) server_negotiation.cc:407] Connection header received
0810 19:01:20.609340 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.609342 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.609401 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.609481 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:20.610240 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.610992 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.612017 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.612379 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.612493 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.613828 (+ 1335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:20.613831 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:20.613833 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:20.613868 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:20.615704 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.616191 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.616195 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.616196 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.616277 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.616594 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.616596 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.616597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.616806 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:20.617011 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.617131 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.617256 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":90,"threads_started":1}
I20260810 19:01:20.617662 18482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.604925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:56303 (local address 127.27.147.190:41613)
0810 19:01:20.605071 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.605076 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.605870 (+ 794us) server_negotiation.cc:407] Connection header received
0810 19:01:20.607194 (+ 1324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.607198 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.607257 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.607364 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:20.608699 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.609515 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.610919 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.611197 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.611305 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.613791 (+ 2486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:20.613795 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:20.613798 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:20.613823 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:20.615304 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.615924 (+ 620us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.615928 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.615930 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.616015 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:20.616409 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:20.616413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:20.616415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:20.616634 (+ 219us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:20.616716 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.617327 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.617452 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:01:20.618405 18948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.605287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:56303)
0810 19:01:20.605720 (+ 433us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.605761 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.606988 (+ 1227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.607509 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.607516 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.607897 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:20.608554 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.608570 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.609652 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.609654 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.610760 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.610772 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.611348 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.611351 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.611491 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.612072 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:20.612091 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:20.613631 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:20.615440 (+ 1809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.615448 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.615463 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.615822 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.616112 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:20.616114 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:20.616116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:20.616312 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:20.616748 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:20.616754 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:20.617047 (+ 293us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.617253 (+ 206us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.617567 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":116,"threads_started":1}
I20260810 19:01:20.619621 18943 heartbeater.cc:347] Connected to a master server at 127.27.147.190:41613
I20260810 19:01:20.619815 18943 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:20.620216 18943 heartbeater.cc:511] Master 127.27.147.190:41613 requested a full tablet report, sending...
I20260810 19:01:20.621140 18149 ts_manager.cc:194] Registered new tserver with Master: f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889)
I20260810 19:01:20.621640 18149 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:56303
I20260810 19:01:20.630992 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:01:20.635160 18949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.631598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60662 (local address 127.27.147.190:41613)
0810 19:01:20.631708 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.631712 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.631928 (+ 216us) server_negotiation.cc:407] Connection header received
0810 19:01:20.632090 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.632092 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.632177 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.632298 (+ 121us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:01:20.632547 (+ 249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.633023 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.633868 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.634080 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.634161 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.634419 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:01:20.634740 (+ 321us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:01:20.634897 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.634952 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.635000 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260810 19:01:20.635761 18149 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:60662:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:01:20.638928 18149 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:01:20.647058 18359 tablet_service.cc:1467] Processing CreateTablet for tablet 674c5167902a4b95b585340dbf171b3f (DEFAULT_TABLE table=test-table [id=51c9ebc0de244df5993d8b50438cfdf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:20.647768 18359 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674c5167902a4b95b585340dbf171b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:20.650669 18286 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap starting.
I20260810 19:01:20.651618 18286 tablet_bootstrap.cc:654] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:20.651714 18960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.646023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:35673 (local address 127.0.0.1:42822)
0810 19:01:20.646300 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.646314 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.646401 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.647555 (+ 1154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.647558 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.647576 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:20.647664 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.647670 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.648730 (+ 1060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.648735 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.650106 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.650117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.651224 (+ 1107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.651228 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.651336 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.651383 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.651464 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.651556 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":69,"thread_start_us":116,"threads_started":1}
I20260810 19:01:20.652222 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.645722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:33889 (local address 127.0.0.1:46368)
0810 19:01:20.645834 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.645851 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.645927 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.647253 (+ 1326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.647256 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.647273 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:20.647363 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.647369 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.648639 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.648643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.650108 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.650117 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.651840 (+ 1723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.651846 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.651949 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.651971 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.652024 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.652109 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":132}
I20260810 19:01:20.652573 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.646297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.27.147.130:35673)
0810 19:01:20.647015 (+ 718us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.647026 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.647046 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:01:20.647108 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.647111 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.647406 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.647583 (+ 177us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:20.647747 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.648553 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.650251 (+ 1698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.651097 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.651219 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.651342 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.652111 (+ 769us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.652208 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":135,"threads_started":1}
I20260810 19:01:20.652845 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.646315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46368 (local address 127.27.147.131:33889)
0810 19:01:20.646733 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.646745 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.646768 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:01:20.646882 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.646888 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.647090 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.647308 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:20.647480 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.648485 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.650758 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.651699 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.652435 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.652558 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.652612 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.652703 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":135,"threads_started":1}
I20260810 19:01:20.653980 18286 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: No bootstrap required, opened a new log
I20260810 19:01:20.654074 18286 ts_tablet_manager.cc:1397] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260810 19:01:20.654505 18286 raft_consensus.cc:348] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.654632 18286 raft_consensus.cc:374] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:20.654672 18286 raft_consensus.cc:729] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286f8b8e33c4427e886d8b0765bfce65, State: Initialized, Role: FOLLOWER
I20260810 19:01:20.654831 18286 consensus_queue.cc:260] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.655113 18286 ts_tablet_manager.cc:1428] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 19:01:20.655601 18816 tablet_service.cc:1467] Processing CreateTablet for tablet 674c5167902a4b95b585340dbf171b3f (DEFAULT_TABLE table=test-table [id=51c9ebc0de244df5993d8b50438cfdf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:20.655615 18610 tablet_service.cc:1467] Processing CreateTablet for tablet 674c5167902a4b95b585340dbf171b3f (DEFAULT_TABLE table=test-table [id=51c9ebc0de244df5993d8b50438cfdf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:20.656621 18816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674c5167902a4b95b585340dbf171b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:20.656672 18610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674c5167902a4b95b585340dbf171b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:20.663477 18966 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Bootstrap starting.
I20260810 19:01:20.664081 18965 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Bootstrap starting.
I20260810 19:01:20.665947 18966 tablet_bootstrap.cc:654] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:20.666317 18965 tablet_bootstrap.cc:654] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:20.666697 18966 log.cc:824] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:20.667042 18965 log.cc:824] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:20.668545 18966 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: No bootstrap required, opened a new log
I20260810 19:01:20.668737 18966 ts_tablet_manager.cc:1397] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 19:01:20.669090 18965 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: No bootstrap required, opened a new log
I20260810 19:01:20.669281 18965 ts_tablet_manager.cc:1397] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 19:01:20.671535 18966 raft_consensus.cc:348] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.671725 18966 raft_consensus.cc:374] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:20.671787 18966 raft_consensus.cc:729] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7bae4f49a064686b780dccd8e364c70, State: Initialized, Role: FOLLOWER
I20260810 19:01:20.672173 18966 consensus_queue.cc:260] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.672104 18965 raft_consensus.cc:348] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.672339 18965 raft_consensus.cc:374] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:20.672415 18965 raft_consensus.cc:729] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13769f01eaff4755b2509c0ff701062c, State: Initialized, Role: FOLLOWER
I20260810 19:01:20.672821 18943 heartbeater.cc:503] Master 127.27.147.190:41613 was elected leader, sending a full tablet report...
I20260810 19:01:20.672839 18965 consensus_queue.cc:260] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.673137 18966 ts_tablet_manager.cc:1428] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 19:01:20.673693 18721 heartbeater.cc:503] Master 127.27.147.190:41613 was elected leader, sending a full tablet report...
I20260810 19:01:20.673812 18965 ts_tablet_manager.cc:1428] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
W20260810 19:01:20.685853 18481 tablet.cc:2369] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:01:20.691352 18494 raft_consensus.cc:482] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:20.691504 18494 raft_consensus.cc:504] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.692437 18494 leader_election.cc:288] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889), 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
I20260810 19:01:20.696151 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.692685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:33889 (local address 127.27.147.129:37959)
0810 19:01:20.692802 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.692819 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.692922 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.693262 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.693266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.693285 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:20.693434 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.693440 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.694184 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.694189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.695026 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.695036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.695686 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.695690 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.695812 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.695849 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.695943 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.696022 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260810 19:01:20.696151 18963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.692771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:37959 (local address 127.27.147.131:33889)
0810 19:01:20.692905 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.692912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.692933 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:01:20.693101 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.693109 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.693149 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.693245 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:20.693555 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.694065 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.695162 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.695559 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.695690 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.695743 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.695962 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.696025 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:01:20.696341 18962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.692790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:55403 (local address 127.27.147.130:35673)
0810 19:01:20.692899 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.692903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.693222 (+ 319us) server_negotiation.cc:407] Connection header received
0810 19:01:20.693370 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.693373 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.693412 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.693494 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:20.693771 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.694249 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.695387 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.695791 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.695919 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.695970 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.696160 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.696228 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:01:20.696367 18974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.692685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:35673 (local address 127.27.147.129:55403)
0810 19:01:20.693135 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.693145 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:20.693249 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:20.693509 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:20.693512 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:20.693528 (+ 16us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:20.693660 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.693666 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.694373 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.694379 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.695263 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:20.695273 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.695924 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.695927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:20.696041 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.696058 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 19:01:20.696145 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 19:01:20.696212 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"mutex_wait_us":152,"thread_start_us":103,"threads_started":1}
I20260810 19:01:20.696808 18854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5167902a4b95b585340dbf171b3f" candidate_uuid: "286f8b8e33c4427e886d8b0765bfce65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7bae4f49a064686b780dccd8e364c70" is_pre_election: true
I20260810 19:01:20.696864 18642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5167902a4b95b585340dbf171b3f" candidate_uuid: "286f8b8e33c4427e886d8b0765bfce65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13769f01eaff4755b2509c0ff701062c" is_pre_election: true
I20260810 19:01:20.697103 18854 raft_consensus.cc:2454] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286f8b8e33c4427e886d8b0765bfce65 in term 0.
I20260810 19:01:20.697119 18642 raft_consensus.cc:2454] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 286f8b8e33c4427e886d8b0765bfce65 in term 0.
I20260810 19:01:20.697602 18279 leader_election.cc:302] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286f8b8e33c4427e886d8b0765bfce65, f7bae4f49a064686b780dccd8e364c70; no voters:
I20260810 19:01:20.697912 18494 raft_consensus.cc:2781] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:01:20.698007 18494 raft_consensus.cc:482] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:01:20.698060 18494 raft_consensus.cc:3037] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:20.699144 18494 raft_consensus.cc:504] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.699513 18494 leader_election.cc:288] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 1 election: Requested vote from peers f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889), 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
I20260810 19:01:20.699841 18854 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5167902a4b95b585340dbf171b3f" candidate_uuid: "286f8b8e33c4427e886d8b0765bfce65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f7bae4f49a064686b780dccd8e364c70"
I20260810 19:01:20.699873 18642 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674c5167902a4b95b585340dbf171b3f" candidate_uuid: "286f8b8e33c4427e886d8b0765bfce65" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13769f01eaff4755b2509c0ff701062c"
I20260810 19:01:20.699977 18854 raft_consensus.cc:3037] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:20.699995 18642 raft_consensus.cc:3037] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:20.701005 18854 raft_consensus.cc:2454] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286f8b8e33c4427e886d8b0765bfce65 in term 1.
I20260810 19:01:20.701067 18642 raft_consensus.cc:2454] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 286f8b8e33c4427e886d8b0765bfce65 in term 1.
I20260810 19:01:20.701327 18279 leader_election.cc:302] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 286f8b8e33c4427e886d8b0765bfce65, f7bae4f49a064686b780dccd8e364c70; no voters:
I20260810 19:01:20.701566 18494 raft_consensus.cc:2781] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:20.701800 18494 raft_consensus.cc:686] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 LEADER]: Becoming Leader. State: Replica: 286f8b8e33c4427e886d8b0765bfce65, State: Running, Role: LEADER
I20260810 19:01:20.702060 18494 consensus_queue.cc:237] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:20.705173 18149 catalog_manager.cc:5495] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 reported cstate change: term changed from 0 to 1, leader changed from <none> to 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129). New cstate: current_term: 1 leader_uuid: "286f8b8e33c4427e886d8b0765bfce65" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } health_report { overall_health: HEALTHY } } }
I20260810 19:01:20.722649 18497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.718998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.27.147.129:44829)
0810 19:01:20.719135 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:20.719142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:20.719158 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:01:20.719369 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:20.719372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:20.719427 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:20.719544 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:01:20.719780 (+ 236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.720321 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.721182 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:20.721428 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:20.721541 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:20.721821 (+ 280us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:01:20.722225 (+ 404us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:01:20.722360 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:20.722443 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 19:01:20.722503 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:01:20.726426 18854 raft_consensus.cc:1261] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 1 FOLLOWER]: Refusing update from remote peer 286f8b8e33c4427e886d8b0765bfce65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:20.726428 18642 raft_consensus.cc:1261] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Refusing update from remote peer 286f8b8e33c4427e886d8b0765bfce65: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:20.727182 18494 consensus_queue.cc:1035] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:20.727406 18975 consensus_queue.cc:1035] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [LEADER]: Connected to new peer: Peer: permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:20.736232 18982 mvcc.cc:205] Tried to move back new op lower bound from 7317047217049137152 to 7317047216958156800. Current Snapshot: MvccSnapshot[applied={T|T < 7317047217049137152}]
I20260810 19:01:20.737432 18983 mvcc.cc:205] Tried to move back new op lower bound from 7317047217049137152 to 7317047216958156800. Current Snapshot: MvccSnapshot[applied={T|T < 7317047217049137152}]
I20260810 19:01:20.742547 18151 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:60662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:20.742704 18151 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:60662:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:20.744978 18151 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Sending DeleteTablet for 3 replicas of tablet 674c5167902a4b95b585340dbf171b3f
I20260810 19:01:20.745606 18816 tablet_service.cc:1514] Processing DeleteTablet for tablet 674c5167902a4b95b585340dbf171b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:20 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:46368
I20260810 19:01:20.745774 18610 tablet_service.cc:1514] Processing DeleteTablet for tablet 674c5167902a4b95b585340dbf171b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:20 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:42822
I20260810 19:01:20.745850 18359 tablet_service.cc:1514] Processing DeleteTablet for tablet 674c5167902a4b95b585340dbf171b3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:20 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:50702
I20260810 19:01:20.746174 18994 tablet_replica.cc:330] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: stopping tablet replica
I20260810 19:01:20.746268 18994 raft_consensus.cc:2229] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:01:20.746268 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 18222
I20260810 19:01:20.754184 18993 tablet_replica.cc:330] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: stopping tablet replica
I20260810 19:01:20.754330 18995 tablet_replica.cc:330] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: stopping tablet replica
I20260810 19:01:20.754550 18993 raft_consensus.cc:2229] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:20.754652 18995 raft_consensus.cc:2229] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:20.754921 18993 raft_consensus.cc:2258] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:20.754973 18995 raft_consensus.cc:2258] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:20.756501 18995 ts_tablet_manager.cc:1905] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:20.759971 18993 ts_tablet_manager.cc:1905] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:20.760087 18995 ts_tablet_manager.cc:1918] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:20.760195 18995 log.cc:1196] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/674c5167902a4b95b585340dbf171b3f
I20260810 19:01:20.760555 18995 ts_tablet_manager.cc:1939] T 674c5167902a4b95b585340dbf171b3f P 13769f01eaff4755b2509c0ff701062c: Deleting consensus metadata
I20260810 19:01:20.761950 18126 catalog_manager.cc:4841] TS 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673): tablet 674c5167902a4b95b585340dbf171b3f (table test-table [id=51c9ebc0de244df5993d8b50438cfdf0]) successfully deleted
I20260810 19:01:20.762372 18993 ts_tablet_manager.cc:1918] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:20.762497 18993 log.cc:1196] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/674c5167902a4b95b585340dbf171b3f
I20260810 19:01:20.762912 18993 ts_tablet_manager.cc:1939] T 674c5167902a4b95b585340dbf171b3f P f7bae4f49a064686b780dccd8e364c70: Deleting consensus metadata
I20260810 19:01:20.764060 18124 catalog_manager.cc:4841] TS f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889): tablet 674c5167902a4b95b585340dbf171b3f (table test-table [id=51c9ebc0de244df5993d8b50438cfdf0]) successfully deleted
W20260810 19:01:20.764349 18126 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:44829 (error 108)
I20260810 19:01:20.765172 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 18492
I20260810 19:01:20.765470 18496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:20.765154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44829 (local address 127.0.0.1:50706)
0810 19:01:20.765295 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:20.765373 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44829: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 19:01:20.767637 18126 catalog_manager.cc:4584] TS 286f8b8e33c4427e886d8b0765bfce65 (127.27.147.129:44829): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 674c5167902a4b95b585340dbf171b3f: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44829: connect: Connection refused (error 111)
I20260810 19:01:20.781206 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 18724
I20260810 19:01:20.791210 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 18095
I20260810 19:01:20.801637 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41613
--webserver_interface=127.27.147.190
--webserver_port=45505
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:41613
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:20.914474 19000 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:20.914698 19000 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:20.914776 19000 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:20.918721 19000 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:20.918807 19000 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:20.918874 19000 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:20.918911 19000 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:20.918941 19000 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:20.928936 19000 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:41613
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:41613
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=45505
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19000
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:20.930054 19000 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:20.931458 19000 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:20.942152 19008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.942382 19015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:20.942152 19009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:20.943470 19000 server_base.cc:1034] running on GCE node
I20260810 19:01:20.944166 19000 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:20.947050 19000 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:20.948247 19000 hybrid_clock.cc:648] HybridClock initialized: now 1786388480948226 us; error 38 us; skew 500 ppm
Aug 10 19:01:20 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388480, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:20.951926 19000 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:20.953099 19000 webserver.cc:460] Webserver started at http://127.27.147.190:45505/ using document root <none> and password file <none>
I20260810 19:01:20.953872 19000 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:20.953981 19000 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:20.959537 19000 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:01:20.962411 19027 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:20.963761 19000 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:01:20.963907 19000 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "56773a037d81472484f5dc475a2742f8"
format_stamp: "Formatted at 2026-08-10 19:01:18 on dist-test-slave-znh6"
server_key: "b447cfcdbd530d55a2ddd24d9988c66d"
server_key_iv: "78f02d729a73e0d582716ce436d17c0c"
server_key_version: "encryptionkey@0"
I20260810 19:01:20.964457 19000 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:20.980957 19000 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:20.983640 19000 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:20.983865 19000 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:20.991034 19000 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:41613
I20260810 19:01:20.991386 19123 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:41613 every 8 connection(s)
I20260810 19:01:20.992120 19000 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:20.996356 19124 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:01:20.997993 19124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Bootstrap starting.
I20260810 19:01:20.999039 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 19000
I20260810 19:01:20.999486 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:44829
--local_ip_for_outbound_sockets=127.27.147.129
--tserver_master_addrs=127.27.147.190:41613
--webserver_port=46327
--webserver_interface=127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:21.017705 19124 log.cc:824] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:21.028932 19124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:01:21.029291 19124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8: Bootstrap complete.
I20260810 19:01:21.033324 19124 raft_consensus.cc:348] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:21.033754 19124 raft_consensus.cc:729] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56773a037d81472484f5dc475a2742f8, State: Initialized, Role: FOLLOWER
I20260810 19:01:21.034224 19124 consensus_queue.cc:260] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:21.034380 19124 raft_consensus.cc:388] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:21.034464 19124 raft_consensus.cc:482] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:21.034590 19124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 2 FOLLOWER]: Advancing to term 3
I20260810 19:01:21.036417 19124 raft_consensus.cc:504] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:21.036747 19124 leader_election.cc:302] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 56773a037d81472484f5dc475a2742f8; no voters:
I20260810 19:01:21.037088 19124 leader_election.cc:288] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 19:01:21.037215 19137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 19:01:21.037472 19137 raft_consensus.cc:686] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [term 3 LEADER]: Becoming Leader. State: Replica: 56773a037d81472484f5dc475a2742f8, State: Running, Role: LEADER
I20260810 19:01:21.037766 19137 consensus_queue.cc:237] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } }
I20260810 19:01:21.038332 19124 sys_catalog.cc:564] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:21.038686 19137 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 56773a037d81472484f5dc475a2742f8. Latest consensus state: current_term: 3 leader_uuid: "56773a037d81472484f5dc475a2742f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } } }
I20260810 19:01:21.038805 19137 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:21.039104 19140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "56773a037d81472484f5dc475a2742f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "56773a037d81472484f5dc475a2742f8" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 41613 } } }
I20260810 19:01:21.039211 19140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 56773a037d81472484f5dc475a2742f8 [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:21.039304 19142 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:21.044472 19142 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:21.045136 19142 catalog_manager.cc:1259] Loaded cluster ID: 8d79316e09024497982c62942722d8ee
I20260810 19:01:21.045200 19142 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 19:01:21.046305 19142 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7ff65a2ca9ad google::LogMessage::Fail() at ??:0
@ 0x7ff65a2cd88a google::LogMessage::SendToLog() at ??:0
@ 0x7ff65a2ca4de google::LogMessage::Flush() at ??:0
@ 0x7ff65a2cba19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7ff664c09fdd _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7ff664c0ad2f kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7ff664c07579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7ff664c4b640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff65c51c3d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7ff65c51cc83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7ff65c51e3f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7ff65c50d512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7ff65db516db start_thread at ??:0
@ 0x7ff658dc861f clone at ??:0
W20260810 19:01:21.147667 19132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:21.147995 19132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:21.148056 19132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:21.153340 19132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:21.153419 19132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:21.153513 19132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:01:21.158684 19132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:44829
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=46327
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:21.159701 19132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:21.160681 19132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:21.170497 19200 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:21.170501 19198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:21.170907 19202 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:21.171052 19132 server_base.cc:1034] running on GCE node
I20260810 19:01:21.171435 19132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:21.172067 19132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:21.173250 19132 hybrid_clock.cc:648] HybridClock initialized: now 1786388481173231 us; error 39 us; skew 500 ppm
Aug 10 19:01:21 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:21.176153 19132 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:01:21.177028 19132 webserver.cc:460] Webserver started at http://127.27.147.129:46327/ using document root <none> and password file <none>
I20260810 19:01:21.177580 19132 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:21.177675 19132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:21.181564 19132 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:01:21.183842 19212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:21.185134 19132 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:01:21.185278 19132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "286f8b8e33c4427e886d8b0765bfce65"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "91ec4736df952b00358fc41daa08745c"
server_key_iv: "f215bc9ddd8ab030422916931bca7f08"
server_key_version: "encryptionkey@0"
I20260810 19:01:21.185796 19132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:21.198915 19132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:21.201972 19132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:21.202159 19132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:21.202690 19132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:21.203821 19225 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 19:01:21.209218 19132 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 19:01:21.209312 19132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.006s user 0.000s sys 0.000s
I20260810 19:01:21.209408 19132 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 19:01:21.211138 19132 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 19:01:21.211216 19132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:21.211501 19225 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap starting.
I20260810 19:01:21.222168 19132 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:44829
I20260810 19:01:21.222386 19411 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:44829 every 8 connection(s)
I20260810 19:01:21.223309 19132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:01:21.227555 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 19132
I20260810 19:01:21.227969 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:35673
--local_ip_for_outbound_sockets=127.27.147.130
--tserver_master_addrs=127.27.147.190:41613
--webserver_port=33525
--webserver_interface=127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:21.234508 19225 log.cc:824] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:21.243477 19225 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:01:21.243904 19225 tablet_bootstrap.cc:492] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Bootstrap complete.
I20260810 19:01:21.245471 19225 ts_tablet_manager.cc:1397] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Time spent bootstrapping tablet: real 0.034s user 0.022s sys 0.009s
I20260810 19:01:21.248677 19225 raft_consensus.cc:348] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:21.249239 19225 raft_consensus.cc:729] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 286f8b8e33c4427e886d8b0765bfce65, State: Initialized, Role: FOLLOWER
I20260810 19:01:21.249778 19225 consensus_queue.cc:260] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:21.250809 19225 ts_tablet_manager.cc:1428] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65: Time spent starting tablet: real 0.005s user 0.007s sys 0.001s
W20260810 19:01:21.283878 19418 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:01:21.225174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:38257)
0810 19:01:21.225659 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.225710 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:21.226510 (+ 800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:21.282918 (+ 56408us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":160,"threads_started":1}
W20260810 19:01:21.284482 19415 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:41613 (0 consecutive failures): Network error: Failed to ping master at 127.27.147.190:41613: Client connection negotiation failed: client connection to 127.27.147.190:41613: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 19:01:21.285246 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.284961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:39853)
0810 19:01:21.285106 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.285179 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 19:01:21.286006 19418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.285772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:57367)
0810 19:01:21.285893 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.285946 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:01:21.286217 19415 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:01:21.344349 19419 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:21.344573 19419 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:21.344614 19419 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:21.348587 19419 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:21.348635 19419 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:21.348730 19419 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:01:21.354204 19419 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:35673
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=33525
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19419
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:21.355227 19419 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:21.356194 19419 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:21.365676 19433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:21.365689 19436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:21.365688 19430 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:21.365867 19419 server_base.cc:1034] running on GCE node
I20260810 19:01:21.366468 19419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:21.367115 19419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:21.368341 19419 hybrid_clock.cc:648] HybridClock initialized: now 1786388481368322 us; error 32 us; skew 500 ppm
Aug 10 19:01:21 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:21.371078 19419 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:01:21.371940 19419 webserver.cc:460] Webserver started at http://127.27.147.130:33525/ using document root <none> and password file <none>
I20260810 19:01:21.372490 19419 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:21.372583 19419 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:21.376200 19419 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:01:21.378185 19446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:21.379410 19419 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:21.379545 19419 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "13769f01eaff4755b2509c0ff701062c"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "9afa5a0b7607f191671593afbd858cbf"
server_key_iv: "92a3c62f256a9c008c5e14afb14c5635"
server_key_version: "encryptionkey@0"
I20260810 19:01:21.380062 19419 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:21.395372 19419 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:21.398118 19419 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:21.398355 19419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:21.398876 19419 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:21.399663 19419 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:21.399737 19419 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:21.399801 19419 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:21.399853 19419 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:21.409693 19419 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:35673
I20260810 19:01:21.409798 19632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:35673 every 8 connection(s)
I20260810 19:01:21.411043 19419 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:01:21.415195 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 19419
I20260810 19:01:21.415619 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:33889
--local_ip_for_outbound_sockets=127.27.147.131
--tserver_master_addrs=127.27.147.190:41613
--webserver_port=34331
--webserver_interface=127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43499
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:21.415668 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.413213 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:58633)
0810 19:01:21.413776 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.413937 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":123,"threads_started":1}
W20260810 19:01:21.416105 19634 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:41613 (0 consecutive failures): Network error: Failed to ping master at 127.27.147.190:41613: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
I20260810 19:01:21.416846 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.416572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:40327)
0810 19:01:21.416705 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.416767 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 19:01:21.417606 19636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.417328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:37085)
0810 19:01:21.417478 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.417531 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 19:01:21.418323 19634 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:01:21.560182 19637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:21.560462 19637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:21.560547 19637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:21.564577 19637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:21.564651 19637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:21.564752 19637 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:01:21.569882 19637 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43499
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:33889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=34331
--tserver_master_addrs=127.27.147.190:41613
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19637
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:21.570947 19637 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:21.571969 19637 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:21.581749 19648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:21.581758 19643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:21.581759 19646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:21.581977 19637 server_base.cc:1034] running on GCE node
I20260810 19:01:21.582398 19637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:21.583166 19637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:21.584324 19637 hybrid_clock.cc:648] HybridClock initialized: now 1786388481584309 us; error 27 us; skew 500 ppm
Aug 10 19:01:21 dist-test-slave-znh6 krb5kdc[17148](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:21.587208 19637 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:01:21.588217 19637 webserver.cc:460] Webserver started at http://127.27.147.131:34331/ using document root <none> and password file <none>
I20260810 19:01:21.588807 19637 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:21.588903 19637 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:21.592779 19637 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260810 19:01:21.594759 19657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:21.596078 19637 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:21.596210 19637 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "f7bae4f49a064686b780dccd8e364c70"
format_stamp: "Formatted at 2026-08-10 19:01:19 on dist-test-slave-znh6"
server_key: "5e182be7ed83d6757292406433d4897d"
server_key_iv: "cb95d280e6b4059b5810120d8e62bd94"
server_key_version: "encryptionkey@0"
I20260810 19:01:21.596782 19637 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:21.614418 19637 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:21.617292 19637 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:21.617480 19637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:21.618037 19637 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:21.618891 19637 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:21.618957 19637 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:21.619028 19637 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:21.619069 19637 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:21.627933 19637 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:33889
I20260810 19:01:21.627986 19849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:33889 every 8 connection(s)
I20260810 19:01:21.628872 19637 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:01:21.632618 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.630888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:52285)
0810 19:01:21.631376 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.631486 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":268,"thread_start_us":108,"threads_started":1}
W20260810 19:01:21.633062 19850 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:41613 (0 consecutive failures): Network error: Failed to ping master at 127.27.147.190:41613: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
I20260810 19:01:21.633667 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.633437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:40333)
0810 19:01:21.633548 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.633598 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 19:01:21.634343 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:21.634129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:45645)
0810 19:01:21.634235 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:21.634287 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260810 19:01:21.634528 19850 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:01:21.637328 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 19637
I20260810 19:01:22.287226 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:22.286772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:60771)
0810 19:01:22.287062 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:22.287114 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":222,"thread_start_us":124,"threads_started":1}
I20260810 19:01:22.419268 19858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:22.418874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:47291)
0810 19:01:22.419123 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:22.419175 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":74,"threads_started":1}
I20260810 19:01:22.479429 19859 raft_consensus.cc:482] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:22.479628 19859 raft_consensus.cc:504] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:22.480674 19859 leader_election.cc:288] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889), 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
Aug 10 19:01:22 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.131@KRBTEST.COM
Aug 10 19:01:22 dist-test-slave-znh6 krb5kdc[17148](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388481, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.130@KRBTEST.COM
I20260810 19:01:22.504236 19860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:22.480912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:35673 (local address 127.27.147.129:54861)
0810 19:01:22.481224 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:22.481237 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:22.481313 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:22.482931 (+ 1618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:22.482937 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:22.483492 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:22.483952 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:22.483967 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.485297 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.485301 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:22.486204 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:22.486214 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.486964 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.486967 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:22.487230 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:22.497254 (+ 10024us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:22.497268 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:22.499442 (+ 2174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:22.501755 (+ 2313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:22.501758 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:22.501759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:22.501996 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:22.502370 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:22.502373 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:22.502374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:22.502503 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:22.503220 (+ 717us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:22.503226 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:22.503479 (+ 253us) client_negotiation.cc:769] Sending connection context
0810 19:01:22.503676 (+ 197us) client_negotiation.cc:240] Negotiation successful
0810 19:01:22.504029 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":98,"spinlock_wait_cycles":18304,"thread_start_us":66,"threads_started":1}
I20260810 19:01:22.504245 19865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:22.481082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:60655 (local address 127.27.147.131:33889)
0810 19:01:22.481511 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:22.481524 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:22.481562 (+ 38us) server_negotiation.cc:407] Connection header received
0810 19:01:22.482205 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:22.482211 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:22.482587 (+ 376us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:22.482923 (+ 336us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:22.484160 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.485186 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.486336 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.486787 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.487017 (+ 230us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:22.499330 (+ 12313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:22.499339 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:22.499360 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:22.499410 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:22.501424 (+ 2014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:22.502091 (+ 667us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:22.502098 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:22.502107 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:22.502237 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:22.502600 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:22.502606 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:22.502610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:22.503100 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:22.503259 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:22.503791 (+ 532us) server_negotiation.cc:299] Negotiation successful
0810 19:01:22.504051 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":96,"threads_started":1}
I20260810 19:01:22.504279 19856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:22.480902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:33889 (local address 127.27.147.129:60655)
0810 19:01:22.481105 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:22.481121 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:22.481224 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:22.482933 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:22.482938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:22.483522 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:22.483958 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:22.483970 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.485342 (+ 1372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.485345 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:22.486206 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:22.486216 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.486969 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.486973 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:22.487228 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:22.497248 (+ 10020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:22.497266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:22.499037 (+ 1771us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:22.501577 (+ 2540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:22.501585 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:22.501599 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:22.501972 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:22.502344 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:22.502347 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:22.502349 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:22.502497 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:22.503223 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:22.503229 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:22.503480 (+ 251us) client_negotiation.cc:769] Sending connection context
0810 19:01:22.503675 (+ 195us) client_negotiation.cc:240] Negotiation successful
0810 19:01:22.504117 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":338,"spinlock_wait_cycles":20736}
I20260810 19:01:22.504271 19861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:22.481082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:54861 (local address 127.27.147.130:35673)
0810 19:01:22.481511 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:22.481530 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:22.481564 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:01:22.482212 (+ 648us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:22.482218 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:22.482547 (+ 329us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:22.482881 (+ 334us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:22.484117 (+ 1236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.485156 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.486335 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:22.486787 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:22.487018 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:22.499722 (+ 12704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:22.499729 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:22.499746 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:22.499786 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:22.501641 (+ 1855us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:22.502129 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:22.502134 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:22.502138 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:22.502245 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:22.502633 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:22.502648 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:22.502649 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:22.503099 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:22.503260 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:22.503790 (+ 530us) server_negotiation.cc:299] Negotiation successful
0810 19:01:22.504084 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":333,"thread_start_us":93,"threads_started":1}
W20260810 19:01:22.507184 19216 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
W20260810 19:01:22.507426 19215 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
I20260810 19:01:22.507505 19215 leader_election.cc:302] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286f8b8e33c4427e886d8b0765bfce65; no voters: 13769f01eaff4755b2509c0ff701062c, f7bae4f49a064686b780dccd8e364c70
I20260810 19:01:22.507822 19859 raft_consensus.cc:2726] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:01:22.635551 19867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:22.635117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:36037)
0810 19:01:22.635399 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:22.635455 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":84,"threads_started":1}
I20260810 19:01:23.288391 19869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:23.287966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:51379)
0810 19:01:23.288221 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:23.288271 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":88,"threads_started":1}
I20260810 19:01:23.420496 19870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:23.420042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:57159)
0810 19:01:23.420300 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:23.420367 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":96,"threads_started":1}
I20260810 19:01:23.636828 19872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:23.636423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:56641)
0810 19:01:23.636667 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:23.636721 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":79,"threads_started":1}
I20260810 19:01:24.289554 19873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:24.289121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:39975)
0810 19:01:24.289405 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:24.289458 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":109,"threads_started":1}
I20260810 19:01:24.421625 19876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:24.421196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:58533)
0810 19:01:24.421467 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:24.421517 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":115,"threads_started":1}
I20260810 19:01:24.533740 19879 raft_consensus.cc:482] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:24.533922 19879 raft_consensus.cc:504] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:24.534456 19879 leader_election.cc:288] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889), 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
W20260810 19:01:24.535346 19215 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
W20260810 19:01:24.535439 19216 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
I20260810 19:01:24.535528 19216 leader_election.cc:302] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286f8b8e33c4427e886d8b0765bfce65; no voters: 13769f01eaff4755b2509c0ff701062c, f7bae4f49a064686b780dccd8e364c70
I20260810 19:01:24.535693 19879 raft_consensus.cc:2726] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:01:24.637948 19880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:24.637539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:42015)
0810 19:01:24.637791 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:24.637859 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260810 19:01:25.290692 19881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:25.290263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:45245)
0810 19:01:25.290521 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:25.290584 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":76,"threads_started":1}
I20260810 19:01:25.422784 19882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:25.422365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:40087)
0810 19:01:25.422650 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:25.422700 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":88,"threads_started":1}
I20260810 19:01:25.639159 19883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:25.638696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:37917)
0810 19:01:25.639014 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:25.639067 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249,"thread_start_us":83,"threads_started":1}
I20260810 19:01:26.291994 19885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:26.291543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:40621)
0810 19:01:26.291842 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:26.291897 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":228,"thread_start_us":77,"threads_started":1}
I20260810 19:01:26.424032 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:26.423628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:51945)
0810 19:01:26.423882 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:26.423936 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":83,"threads_started":1}
I20260810 19:01:26.640373 19889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:26.639914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:60709)
0810 19:01:26.640221 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:26.640278 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":74,"threads_started":1}
I20260810 19:01:26.704802 19890 raft_consensus.cc:482] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:26.704942 19890 raft_consensus.cc:504] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:26.705336 19890 leader_election.cc:288] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889), 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
W20260810 19:01:26.706135 19215 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
W20260810 19:01:26.706231 19216 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
I20260810 19:01:26.706343 19216 leader_election.cc:302] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286f8b8e33c4427e886d8b0765bfce65; no voters: 13769f01eaff4755b2509c0ff701062c, f7bae4f49a064686b780dccd8e364c70
I20260810 19:01:26.706508 19890 raft_consensus.cc:2726] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:01:27.293182 19891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:27.292766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:58035)
0810 19:01:27.293028 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:27.293084 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260810 19:01:27.425130 19892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:27.424738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:34613)
0810 19:01:27.424990 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:27.425044 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":87,"threads_started":1}
I20260810 19:01:27.641551 19893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:27.641140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:42287)
0810 19:01:27.641400 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:27.641455 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":91,"threads_started":1}
I20260810 19:01:28.294356 19894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:28.293913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:54189)
0810 19:01:28.294202 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:28.294257 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":92,"threads_started":1}
I20260810 19:01:28.426342 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:28.425911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:49319)
0810 19:01:28.426202 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:28.426255 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":225,"thread_start_us":89,"threads_started":1}
I20260810 19:01:28.642710 19896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:28.642261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:39821)
0810 19:01:28.642560 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:28.642617 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":89,"threads_started":1}
I20260810 19:01:29.295586 19897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:29.295187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:42203)
0810 19:01:29.295441 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:29.295492 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":88,"threads_started":1}
I20260810 19:01:29.427501 19899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:29.427112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:57307)
0810 19:01:29.427349 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:29.427406 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":72,"threads_started":1}
I20260810 19:01:29.643787 19902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:29.643383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:37567)
0810 19:01:29.643642 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:29.643694 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":82,"threads_started":1}
I20260810 19:01:30.296690 19903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:30.296280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:49795)
0810 19:01:30.296538 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:30.296592 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":99,"threads_started":1}
I20260810 19:01:30.428619 19904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:30.428189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:41151)
0810 19:01:30.428462 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:30.428526 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":90,"threads_started":1}
I20260810 19:01:30.645013 19906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:30.644553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:41265)
0810 19:01:30.644846 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:30.644900 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":101,"threads_started":1}
I20260810 19:01:31.298060 19908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:31.297613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:34729)
0810 19:01:31.297902 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:31.297961 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":79,"threads_started":1}
I20260810 19:01:31.429790 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:31.429371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:37987)
0810 19:01:31.429641 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:31.429698 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":94,"threads_started":1}
I20260810 19:01:31.492853 19911 raft_consensus.cc:482] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:31.493029 19911 raft_consensus.cc:504] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f7bae4f49a064686b780dccd8e364c70" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 33889 } } peers { permanent_uuid: "13769f01eaff4755b2509c0ff701062c" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 35673 } } peers { permanent_uuid: "286f8b8e33c4427e886d8b0765bfce65" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44829 } }
I20260810 19:01:31.493471 19911 leader_election.cc:288] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889), 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673)
W20260810 19:01:31.494320 19216 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 13769f01eaff4755b2509c0ff701062c (127.27.147.130:35673): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
W20260810 19:01:31.494423 19215 leader_election.cc:341] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f7bae4f49a064686b780dccd8e364c70 (127.27.147.131:33889): Not found: Tablet not found: 674c5167902a4b95b585340dbf171b3f
I20260810 19:01:31.494484 19215 leader_election.cc:302] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 286f8b8e33c4427e886d8b0765bfce65; no voters: 13769f01eaff4755b2509c0ff701062c, f7bae4f49a064686b780dccd8e364c70
I20260810 19:01:31.494637 19911 raft_consensus.cc:2726] T 674c5167902a4b95b585340dbf171b3f P 286f8b8e33c4427e886d8b0765bfce65 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:01:31.646287 19912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:31.645822 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:33573)
0810 19:01:31.646123 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:31.646185 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":91,"threads_started":1}
I20260810 19:01:32.299556 19914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:32.298978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:59143)
0810 19:01:32.299322 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:32.299401 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":115,"threads_started":1}
I20260810 19:01:32.431053 19916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:32.430542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:55821)
0810 19:01:32.430892 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:32.430960 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":97,"threads_started":1}
I20260810 19:01:32.647601 19922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:32.647160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:40203)
0810 19:01:32.647440 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:32.647499 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204,"thread_start_us":102,"threads_started":1}
I20260810 19:01:33.300872 19923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:33.300400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:47605)
0810 19:01:33.300710 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:33.300771 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":129,"threads_started":1}
I20260810 19:01:33.432312 19925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:33.431886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:45363)
0810 19:01:33.432150 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:33.432210 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":187,"thread_start_us":78,"threads_started":1}
I20260810 19:01:33.648756 19927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:33.648354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:49961)
0810 19:01:33.648603 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:33.648658 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":78,"threads_started":1}
I20260810 19:01:34.302043 19928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:34.301617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:50585)
0810 19:01:34.301892 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:34.301946 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":94,"threads_started":1}
I20260810 19:01:34.433583 19929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:34.433167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:58073)
0810 19:01:34.433433 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:34.433487 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197,"thread_start_us":92,"threads_started":1}
I20260810 19:01:34.650034 19930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:34.649525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:46265)
0810 19:01:34.649862 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:34.649922 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":80,"threads_started":1}
I20260810 19:01:35.303345 19932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:35.302909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:47289)
0810 19:01:35.303172 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:35.303234 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":80,"threads_started":1}
I20260810 19:01:35.434753 19933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:35.434308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:48363)
0810 19:01:35.434586 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:35.434648 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":80,"threads_started":1}
I20260810 19:01:35.651345 19934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:35.650937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:53067)
0810 19:01:35.651196 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:35.651248 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":95,"threads_started":1}
I20260810 19:01:36.304610 19935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:36.304222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.129:60929)
0810 19:01:36.304460 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:36.304510 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":73,"threads_started":1}
I20260810 19:01:36.435987 19936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:36.435584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.130:47389)
0810 19:01:36.435820 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:36.435889 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":80,"threads_started":1}
I20260810 19:01:36.644686 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 19132
I20260810 19:01:36.652663 19938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:36.652263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:41613 (local address 127.27.147.131:48431)
0810 19:01:36.652513 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:36.652566 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:41613: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":87,"threads_started":1}
I20260810 19:01:36.658942 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 19419
I20260810 19:01:36.668129 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 19637
2026-08-10T19:01:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (19427 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:01:36 dist-test-slave-znh6 krb5kdc[19946](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:01:36 dist-test-slave-znh6 krb5kdc[19946](info): set up 2 sockets
Aug 10 19:01:36 dist-test-slave-znh6 krb5kdc[19946](info): commencing operation
krb5kdc: starting...
W20260810 19:01:37.924860 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.229s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:01:38.053290 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38471
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:38.167009 19971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:38.167269 19971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:38.167348 19971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:38.171312 19971 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:01:38.171382 19971 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:01:38.171411 19971 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:01:38.171442 19971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:38.171481 19971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:38.171526 19971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:38.171578 19971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:38.171617 19971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:38.176599 19971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:38.177536 19971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:38.178555 19971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:38.186808 19978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:38.186854 19983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:38.186826 19979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:38.186861 19971 server_base.cc:1034] running on GCE node
I20260810 19:01:38.187578 19971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:38.188673 19971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:38.189858 19971 hybrid_clock.cc:648] HybridClock initialized: now 1786388498189831 us; error 39 us; skew 500 ppm
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:38.192497 19971 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:38.193310 19971 webserver.cc:460] Webserver started at http://127.27.147.190:40017/ using document root <none> and password file <none>
I20260810 19:01:38.193790 19971 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:38.193848 19971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:38.194012 19971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:38.195927 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "94013a28caaa4bae854ed59213d2cdbf"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "c450e2634304f39b044c3dd856bf07db"
server_key_iv: "f087c98d5d4d07207e38e4dcfd39c88d"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.196405 19971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "94013a28caaa4bae854ed59213d2cdbf"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "c450e2634304f39b044c3dd856bf07db"
server_key_iv: "f087c98d5d4d07207e38e4dcfd39c88d"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.199853 19971 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 19:01:38.202062 19993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.203311 19971 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:38.203410 19971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "94013a28caaa4bae854ed59213d2cdbf"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "c450e2634304f39b044c3dd856bf07db"
server_key_iv: "f087c98d5d4d07207e38e4dcfd39c88d"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.203488 19971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:38.220083 19971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:38.222637 19971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:38.222774 19971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:38.229784 19971 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38471
I20260810 19:01:38.230067 20074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38471 every 8 connection(s)
I20260810 19:01:38.230686 19971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:38.233305 20076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:38.238584 20076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Bootstrap starting.
I20260810 19:01:38.239405 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 19971
I20260810 19:01:38.239547 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:01:38.239838 28238 external_mini_cluster.cc:1428] Setting key ee7ac849692ed9b12e6617f27c952df1
I20260810 19:01:38.241220 20076 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:38.242115 20076 log.cc:824] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:38.244217 20076 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: No bootstrap required, opened a new log
I20260810 19:01:38.248273 20076 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:38.248503 20076 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:38.248594 20076 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94013a28caaa4bae854ed59213d2cdbf, State: Initialized, Role: FOLLOWER
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:38.249024 20076 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:38.249174 20076 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:38.249243 20076 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:38.249348 20076 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:38.250540 20076 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:38.250968 20076 leader_election.cc:302] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94013a28caaa4bae854ed59213d2cdbf; no voters:
I20260810 19:01:38.251297 20076 leader_election.cc:288] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:01:38.251364 20083 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:38.251579 20083 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 LEADER]: Becoming Leader. State: Replica: 94013a28caaa4bae854ed59213d2cdbf, State: Running, Role: LEADER
I20260810 19:01:38.251909 20083 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:38.252517 20076 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:38.253033 20083 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94013a28caaa4bae854ed59213d2cdbf. Latest consensus state: current_term: 1 leader_uuid: "94013a28caaa4bae854ed59213d2cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } } }
I20260810 19:01:38.253823 20083 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:38.253705 20084 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94013a28caaa4bae854ed59213d2cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } } }
I20260810 19:01:38.254132 20084 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:38.254861 20101 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:38.255597 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.240920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.27.147.190:38471)
0810 19:01:38.241396 (+ 476us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:38.241418 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:38.241476 (+ 58us) server_negotiation.cc:407] Connection header received
0810 19:01:38.242269 (+ 793us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:38.242291 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:38.242666 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:38.243028 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:38.243702 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.245142 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.246074 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.246777 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.246961 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.249705 (+ 2744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:38.249711 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:38.249726 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:38.249754 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:38.251792 (+ 2038us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.252329 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.252334 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.252340 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.252497 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.252810 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.252813 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.252815 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.253330 (+ 515us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:38.254134 (+ 804us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:38.254299 (+ 165us) server_negotiation.cc:299] Negotiation successful
0810 19:01:38.254590 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":316,"thread_start_us":133,"threads_started":1}
I20260810 19:01:38.257831 20101 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:38.263588 20101 catalog_manager.cc:1347] Generated new cluster ID: 653b1fa2af4e4881bb3e9d459f47b404
I20260810 19:01:38.263705 20101 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:38.272307 20101 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:01:38.273145 20101 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:38.279145 20101 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Generated new TSK 0
I20260810 19:01:38.279740 20101 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 19:01:38.317414 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38471
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:38.432446 20125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:38.432709 20125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:38.432780 20125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:38.436735 20125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:38.436816 20125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:38.436902 20125 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:01:38.441936 20125 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:38.442971 20125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:38.443972 20125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:38.453791 20137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:38.453769 20141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:38.453785 20138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:38.454097 20125 server_base.cc:1034] running on GCE node
I20260810 19:01:38.454510 20125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:38.455188 20125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:38.456369 20125 hybrid_clock.cc:648] HybridClock initialized: now 1786388498456343 us; error 37 us; skew 500 ppm
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:38.459111 20125 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:01:38.459862 20125 webserver.cc:460] Webserver started at http://127.27.147.129:34613/ using document root <none> and password file <none>
I20260810 19:01:38.460345 20125 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:38.460408 20125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:38.460570 20125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:38.462397 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "c7ea371eba4948fe8503a51fa8dfec4a"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "2aff3f54c8833f0c3d83abb1398fc430"
server_key_iv: "44c9fb723b5f45789bc290e5e072c3fa"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.462996 20125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "c7ea371eba4948fe8503a51fa8dfec4a"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "2aff3f54c8833f0c3d83abb1398fc430"
server_key_iv: "44c9fb723b5f45789bc290e5e072c3fa"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.466426 20125 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:01:38.468961 20150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.470134 20125 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:01:38.470232 20125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "c7ea371eba4948fe8503a51fa8dfec4a"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "2aff3f54c8833f0c3d83abb1398fc430"
server_key_iv: "44c9fb723b5f45789bc290e5e072c3fa"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.470336 20125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:38.503448 20125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:38.506721 20125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:38.506935 20125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:38.507575 20125 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:38.508351 20125 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:38.508399 20125 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.508438 20125 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:38.508453 20125 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.517248 20125 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:44319
I20260810 19:01:38.517297 20344 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:44319 every 8 connection(s)
I20260810 19:01:38.518096 20125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:01:38.523870 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 20125
I20260810 19:01:38.524016 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:01:38.524354 28238 external_mini_cluster.cc:1428] Setting key 00d5157ee2a9152617a9819b13a5ee1a
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:38.532119 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.519880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:53745 (local address 127.27.147.190:38471)
0810 19:01:38.520047 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:38.520053 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:38.521066 (+ 1013us) server_negotiation.cc:407] Connection header received
0810 19:01:38.522070 (+ 1004us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:38.522074 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:38.522122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:38.522199 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:38.523392 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.524170 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.525361 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.525606 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.525718 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.528445 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:38.528458 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:38.528463 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:38.528495 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:38.529937 (+ 1442us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.530473 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.530475 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.530477 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.530557 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.530956 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.530961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.530963 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.531143 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:38.531257 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:38.531811 (+ 554us) server_negotiation.cc:299] Negotiation successful
0810 19:01:38.531945 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 19:01:38.532842 20348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.520253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:53745)
0810 19:01:38.520915 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:38.520967 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:38.521860 (+ 893us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:38.522356 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:38.522362 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:38.522702 (+ 340us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:38.523254 (+ 552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:38.523269 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.524294 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.524297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:38.525228 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:38.525238 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.525760 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.525764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:38.525936 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.526517 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:38.526532 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:38.528274 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:38.530048 (+ 1774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:38.530052 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:38.530063 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:38.530308 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:38.530657 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:38.530660 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:38.530662 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:38.530799 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:38.531257 (+ 458us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:38.531265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:38.531537 (+ 272us) client_negotiation.cc:769] Sending connection context
0810 19:01:38.531752 (+ 215us) client_negotiation.cc:240] Negotiation successful
0810 19:01:38.532059 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":111,"threads_started":1}
I20260810 19:01:38.534077 20345 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38471
I20260810 19:01:38.534334 20345 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:38.534943 20345 heartbeater.cc:511] Master 127.27.147.190:38471 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 19:01:38.536626 20015 ts_manager.cc:194] Registered new tserver with Master: c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129:44319)
I20260810 19:01:38.537720 20015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:53745
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 19:01:38.554174 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38471
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:38.671510 20352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:38.671744 20352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:38.671785 20352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:38.675889 20352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:38.675940 20352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:38.676036 20352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:01:38.681495 20352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20352
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:38.682579 20352 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:38.683642 20352 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:38.693194 20357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:38.693332 20352 server_base.cc:1034] running on GCE node
W20260810 19:01:38.693205 20363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:38.693194 20358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:38.693948 20352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:38.694506 20352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:38.695676 20352 hybrid_clock.cc:648] HybridClock initialized: now 1786388498695661 us; error 24 us; skew 500 ppm
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:38.698263 20352 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:01:38.699175 20352 webserver.cc:460] Webserver started at http://127.27.147.130:43051/ using document root <none> and password file <none>
I20260810 19:01:38.699692 20352 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:38.699798 20352 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:38.699986 20352 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:38.701977 20352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "c0556cee02b34302b1a53bddb0ff90ce"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "faa86205bf82e41fd9ae6b98619af665"
server_key_iv: "80b3ff7bf30066fe3605c9659c30dbb4"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.702486 20352 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "c0556cee02b34302b1a53bddb0ff90ce"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "faa86205bf82e41fd9ae6b98619af665"
server_key_iv: "80b3ff7bf30066fe3605c9659c30dbb4"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.706068 20352 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 19:01:38.708505 20377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.709676 20352 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:01:38.709806 20352 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "c0556cee02b34302b1a53bddb0ff90ce"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "faa86205bf82e41fd9ae6b98619af665"
server_key_iv: "80b3ff7bf30066fe3605c9659c30dbb4"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.709957 20352 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:38.734191 20352 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:38.737458 20352 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:38.737663 20352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:38.738257 20352 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:38.739074 20352 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:38.739122 20352 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.739192 20352 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:38.739233 20352 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.747924 20352 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:44475
I20260810 19:01:38.747948 20566 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:44475 every 8 connection(s)
I20260810 19:01:38.748790 20352 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:01:38.750697 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 20352
I20260810 19:01:38.750865 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:01:38.751187 28238 external_mini_cluster.cc:1428] Setting key d082482f95a8ce35f38441b24bb0dc4f
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 19:01:38.764472 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.750623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:33591 (local address 127.27.147.190:38471)
0810 19:01:38.750795 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:38.750801 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:38.752364 (+ 1563us) server_negotiation.cc:407] Connection header received
0810 19:01:38.753307 (+ 943us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:38.753312 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:38.753376 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:38.753470 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:38.754779 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.755763 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.757055 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.757272 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.757356 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.760682 (+ 3326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:38.760685 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:38.760687 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:38.760712 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:38.762166 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.762792 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.762795 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.762796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.762959 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.763387 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.763390 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.763391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.763579 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:38.763656 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:38.764128 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 19:01:38.764292 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:01:38.765182 20572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.751041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:33591)
0810 19:01:38.752200 (+ 1159us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:38.752251 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:38.753099 (+ 848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:38.753619 (+ 520us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:38.753625 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:38.754068 (+ 443us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:38.754637 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:38.754650 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.755916 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.755919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:38.756918 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:38.756933 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.757436 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.757443 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:38.757685 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.758303 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:38.758333 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:38.760499 (+ 2166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:38.762302 (+ 1803us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:38.762308 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:38.762322 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:38.762681 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:38.763061 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:38.763068 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:38.763070 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:38.763281 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:38.763695 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:38.763700 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:38.763918 (+ 218us) client_negotiation.cc:769] Sending connection context
0810 19:01:38.764078 (+ 160us) client_negotiation.cc:240] Negotiation successful
0810 19:01:38.764381 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":961,"thread_start_us":613,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
I20260810 19:01:38.766414 20567 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38471
I20260810 19:01:38.766650 20567 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:38.767176 20567 heartbeater.cc:511] Master 127.27.147.190:38471 requested a full tablet report, sending...
I20260810 19:01:38.768272 20015 ts_manager.cc:194] Registered new tserver with Master: c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475)
I20260810 19:01:38.769160 20015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:33591
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 19:01:38.781971 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38471
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:38.897433 20578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:38.897658 20578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:38.897699 20578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:38.901640 20578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:38.901692 20578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:38.901784 20578 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:01:38.906978 20578 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:38.907979 20578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:38.908946 20578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:38.918525 20587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:38.918531 20586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:38.918576 20589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:38.918771 20578 server_base.cc:1034] running on GCE node
I20260810 19:01:38.919185 20578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:38.919783 20578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:38.920977 20578 hybrid_clock.cc:648] HybridClock initialized: now 1786388498920965 us; error 24 us; skew 500 ppm
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:38.923635 20578 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:01:38.924440 20578 webserver.cc:460] Webserver started at http://127.27.147.131:40217/ using document root <none> and password file <none>
I20260810 19:01:38.924989 20578 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:38.925081 20578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:38.925375 20578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:01:38.927423 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "fe573f7086a74540be878a3e249ffc9e"
server_key_iv: "daf0d21456279f20a398ae58d2a7fdff"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.928033 20578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "fe573f7086a74540be878a3e249ffc9e"
server_key_iv: "daf0d21456279f20a398ae58d2a7fdff"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.931607 20578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:01:38.933959 20600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.935132 20578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:01:38.935261 20578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "fe573f7086a74540be878a3e249ffc9e"
server_key_iv: "daf0d21456279f20a398ae58d2a7fdff"
server_key_version: "encryptionkey@0"
I20260810 19:01:38.935370 20578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:38.945452 20578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:38.948231 20578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:38.948419 20578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:38.948984 20578 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:38.949769 20578 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:01:38.949841 20578 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.949910 20578 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:01:38.949962 20578 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:38.959012 20578 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:35945
I20260810 19:01:38.959040 20781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:35945 every 8 connection(s)
I20260810 19:01:38.959836 20578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:01:38.968503 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 20578
I20260810 19:01:38.968631 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
Aug 10 19:01:38 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388498, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:38.968978 28238 external_mini_cluster.cc:1428] Setting key d47d155aac8d6f6a94ada0140eb5d6b4
I20260810 19:01:38.973028 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.961579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:48175 (local address 127.27.147.190:38471)
0810 19:01:38.961727 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:38.961731 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:38.962705 (+ 974us) server_negotiation.cc:407] Connection header received
0810 19:01:38.963547 (+ 842us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:38.963550 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:38.963597 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:38.963670 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:38.964746 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.965265 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.966472 (+ 1207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.966683 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.966762 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.969550 (+ 2788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:38.969553 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:38.969555 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:38.969579 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:38.971260 (+ 1681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.971739 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.971742 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.971743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.971814 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.972114 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.972116 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.972117 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.972278 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:38.972351 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:38.972808 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 19:01:38.972906 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:01:38.973863 20784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.961914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:48175)
0810 19:01:38.962561 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:38.962600 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:38.963359 (+ 759us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:38.963818 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:38.963824 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:38.964171 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:38.964618 (+ 447us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:38.964631 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.965491 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.965494 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:38.966360 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:38.966369 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.966916 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.966921 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:38.967073 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.967670 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:38.967687 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:38.969375 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:38.971382 (+ 2007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:38.971393 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:38.971404 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:38.971654 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:38.971897 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:38.971899 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:38.971900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:38.972034 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:38.972365 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:38.972369 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:38.972593 (+ 224us) client_negotiation.cc:769] Sending connection context
0810 19:01:38.972739 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 19:01:38.972970 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":113,"threads_started":1}
I20260810 19:01:38.975033 20782 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38471
I20260810 19:01:38.975263 20782 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:38.975728 20782 heartbeater.cc:511] Master 127.27.147.190:38471 requested a full tablet report, sending...
I20260810 19:01:38.976716 20015 ts_manager.cc:194] Registered new tserver with Master: a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
I20260810 19:01:38.977273 20015 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:48175
I20260810 19:01:38.982702 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:01:38.991465 20080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:38.984746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48616 (local address 127.27.147.190:38471)
0810 19:01:38.984883 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:38.984888 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:38.985061 (+ 173us) server_negotiation.cc:407] Connection header received
0810 19:01:38.985137 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:38.985139 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:38.985173 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:38.985234 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:38.985846 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.986317 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.987279 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:38.987491 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:38.987569 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:38.988836 (+ 1267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:38.988839 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:38.988841 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:38.988857 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:38.989974 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.990334 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.990336 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.990338 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.990404 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:38.990629 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:38.990631 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:38.990632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:38.990882 (+ 250us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:38.990993 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:38.991212 (+ 219us) server_negotiation.cc:299] Negotiation successful
0810 19:01:38.991302 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:01:38.994040 20015 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48616:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:01:38.997082 20015 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:01:39.011090 20808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.005622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:35945 (local address 127.0.0.1:50766)
0810 19:01:39.006210 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:39.006226 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:39.006307 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:39.007059 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:39.007063 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:39.007077 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:39.007167 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.007173 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.008520 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.008523 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.009484 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.009494 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.010489 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.010495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.010637 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.010687 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 19:01:39.010789 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 19:01:39.010950 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":108,"threads_started":1}
I20260810 19:01:39.011237 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.005757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34416 (local address 127.27.147.130:44475)
0810 19:01:39.006123 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:39.006138 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:39.006160 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:01:39.006245 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:39.006251 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:39.006449 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:39.006611 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:39.006988 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.008171 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.009945 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.010533 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.010701 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.010866 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:39.010976 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 19:01:39.011067 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":134,"threads_started":1}
I20260810 19:01:39.011610 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.005103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41954 (local address 127.27.147.129:44319)
0810 19:01:39.005474 (+ 371us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:39.005491 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:39.005517 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:01:39.006405 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:39.006415 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:39.006604 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:39.006733 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:39.007273 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.008102 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.009916 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.010743 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.010890 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.011023 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:39.011344 (+ 321us) server_negotiation.cc:299] Negotiation successful
0810 19:01:39.011440 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":71,"threads_started":1}
I20260810 19:01:39.011675 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.006068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50766 (local address 127.27.147.131:35945)
0810 19:01:39.006472 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:39.006483 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:39.006506 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:01:39.006580 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:39.006584 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:39.006808 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:39.007079 (+ 271us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:39.007273 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.008333 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.009609 (+ 1276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.010348 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.010472 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.010612 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:39.011465 (+ 853us) server_negotiation.cc:299] Negotiation successful
0810 19:01:39.011541 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":139,"threads_started":1}
I20260810 19:01:39.011219 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.005320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44475 (local address 127.0.0.1:34416)
0810 19:01:39.005666 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:39.005680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:39.005768 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:39.006657 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:39.006666 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:39.006693 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:39.006851 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.006860 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.008420 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.008424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.009783 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.009795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.010694 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.010700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.010835 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.010866 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 19:01:39.010945 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:01:39.011028 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":96,"thread_start_us":90,"threads_started":1}
I20260810 19:01:39.012113 20800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.004905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44319 (local address 127.0.0.1:41954)
0810 19:01:39.005249 (+ 344us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:39.005301 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:39.005438 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:39.006955 (+ 1517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:39.006960 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:39.006980 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:39.007088 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.007094 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.008232 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.008238 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.009783 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.009795 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.011083 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.011086 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.011177 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.011197 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 19:01:39.011875 (+ 678us) client_negotiation.cc:240] Negotiation successful
0810 19:01:39.011974 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":91,"thread_start_us":99,"threads_started":1}
I20260810 19:01:39.014760 20458 tablet_service.cc:1467] Processing CreateTablet for tablet c984e18c807b4f35a0d9138eec97f812 (DEFAULT_TABLE table=test-table [id=ce6c20d7094649dfa93d9a779b688f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:39.014770 20676 tablet_service.cc:1467] Processing CreateTablet for tablet c984e18c807b4f35a0d9138eec97f812 (DEFAULT_TABLE table=test-table [id=ce6c20d7094649dfa93d9a779b688f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:39.014770 20231 tablet_service.cc:1467] Processing CreateTablet for tablet c984e18c807b4f35a0d9138eec97f812 (DEFAULT_TABLE table=test-table [id=ce6c20d7094649dfa93d9a779b688f61]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:01:39.015801 20458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c984e18c807b4f35a0d9138eec97f812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:39.015836 20231 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c984e18c807b4f35a0d9138eec97f812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:39.015843 20676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c984e18c807b4f35a0d9138eec97f812. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:01:39.021852 20811 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Bootstrap starting.
I20260810 19:01:39.023229 20812 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap starting.
I20260810 19:01:39.025024 20811 tablet_bootstrap.cc:654] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:39.025400 20812 tablet_bootstrap.cc:654] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:39.026017 20811 log.cc:824] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:39.026695 20812 log.cc:824] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:39.026777 20813 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Bootstrap starting.
I20260810 19:01:39.028046 20811 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: No bootstrap required, opened a new log
I20260810 19:01:39.028257 20811 ts_tablet_manager.cc:1397] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260810 19:01:39.028823 20812 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: No bootstrap required, opened a new log
I20260810 19:01:39.029042 20812 ts_tablet_manager.cc:1397] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 19:01:39.029315 20813 tablet_bootstrap.cc:654] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Neither blocks nor log segments found. Creating new log.
I20260810 19:01:39.030123 20813 log.cc:824] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:39.031804 20811 raft_consensus.cc:348] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.032058 20811 raft_consensus.cc:374] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:39.032121 20813 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: No bootstrap required, opened a new log
I20260810 19:01:39.032147 20811 raft_consensus.cc:729] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0556cee02b34302b1a53bddb0ff90ce, State: Initialized, Role: FOLLOWER
I20260810 19:01:39.032305 20813 ts_tablet_manager.cc:1397] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 19:01:39.032663 20811 consensus_queue.cc:260] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.032853 20812 raft_consensus.cc:348] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.033159 20812 raft_consensus.cc:374] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:39.033272 20812 raft_consensus.cc:729] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ea371eba4948fe8503a51fa8dfec4a, State: Initialized, Role: FOLLOWER
I20260810 19:01:39.033351 20567 heartbeater.cc:503] Master 127.27.147.190:38471 was elected leader, sending a full tablet report...
I20260810 19:01:39.033720 20811 ts_tablet_manager.cc:1428] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 19:01:39.033780 20812 consensus_queue.cc:260] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.034354 20345 heartbeater.cc:503] Master 127.27.147.190:38471 was elected leader, sending a full tablet report...
I20260810 19:01:39.034686 20812 ts_tablet_manager.cc:1428] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent starting tablet: real 0.006s user 0.006s sys 0.000s
I20260810 19:01:39.037209 20813 raft_consensus.cc:348] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.037513 20813 raft_consensus.cc:374] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:01:39.037575 20813 raft_consensus.cc:729] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88d6cfe7bbd4cf9858a973c3d7a9f00, State: Initialized, Role: FOLLOWER
I20260810 19:01:39.037981 20813 consensus_queue.cc:260] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.038928 20813 ts_tablet_manager.cc:1428] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Time spent starting tablet: real 0.007s user 0.007s sys 0.000s
I20260810 19:01:39.039264 20782 heartbeater.cc:503] Master 127.27.147.190:38471 was elected leader, sending a full tablet report...
W20260810 19:01:39.210796 20783 tablet.cc:2369] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:01:39.222397 20822 raft_consensus.cc:482] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:01:39.222590 20822 raft_consensus.cc:504] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.223716 20822 leader_election.cc:288] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945), c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475)
I20260810 19:01:39.229252 20810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.224052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:49869 (local address 127.27.147.131:35945)
0810 19:01:39.224181 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:39.224186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:39.224541 (+ 355us) server_negotiation.cc:407] Connection header received
0810 19:01:39.224708 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:39.224711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:39.224767 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:39.224852 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:39.225273 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.226071 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.227937 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.228552 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.228677 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.228732 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:39.229052 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 19:01:39.229122 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:01:39.229254 20809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.224052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:58921 (local address 127.27.147.130:44475)
0810 19:01:39.224181 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:39.224186 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:39.224541 (+ 355us) server_negotiation.cc:407] Connection header received
0810 19:01:39.224708 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:39.224711 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:39.224772 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:39.224854 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:01:39.225273 (+ 419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.226076 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.227937 (+ 1861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.228551 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.228677 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.228732 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:39.229052 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 19:01:39.229122 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:01:39.229300 20824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.223970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:35945 (local address 127.27.147.129:49869)
0810 19:01:39.224416 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:39.224433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:39.224589 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:39.224887 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:39.224890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:39.224919 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:39.225143 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.225150 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.226209 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.226213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.227816 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.227822 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.228681 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.228685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.228896 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.228941 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 19:01:39.229020 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:01:39.229115 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"mutex_wait_us":281,"spinlock_wait_cycles":32896,"thread_start_us":105,"threads_started":1}
I20260810 19:01:39.229318 20825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.223970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44475 (local address 127.27.147.129:58921)
0810 19:01:39.224416 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:39.224433 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:39.224584 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:39.224887 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:39.224890 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:39.224918 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:01:39.225142 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.225150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.226209 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.226213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.227798 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:39.227814 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.228681 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.228685 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:39.228872 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.228938 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 19:01:39.229020 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:01:39.229115 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":353,"mutex_wait_us":249,"spinlock_wait_cycles":17408,"thread_start_us":123,"threads_started":1}
I20260810 19:01:39.230156 20490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c984e18c807b4f35a0d9138eec97f812" candidate_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0556cee02b34302b1a53bddb0ff90ce" is_pre_election: true
I20260810 19:01:39.230212 20708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c984e18c807b4f35a0d9138eec97f812" candidate_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" is_pre_election: true
I20260810 19:01:39.230470 20490 raft_consensus.cc:2454] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ea371eba4948fe8503a51fa8dfec4a in term 0.
I20260810 19:01:39.230490 20708 raft_consensus.cc:2454] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c7ea371eba4948fe8503a51fa8dfec4a in term 0.
I20260810 19:01:39.231022 20155 leader_election.cc:302] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce, c7ea371eba4948fe8503a51fa8dfec4a; no voters:
I20260810 19:01:39.231304 20822 raft_consensus.cc:2781] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:01:39.231388 20822 raft_consensus.cc:482] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:01:39.231433 20822 raft_consensus.cc:3037] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:39.232473 20822 raft_consensus.cc:504] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.232831 20822 leader_election.cc:288] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 1 election: Requested vote from peers a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945), c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475)
I20260810 19:01:39.233174 20490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c984e18c807b4f35a0d9138eec97f812" candidate_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0556cee02b34302b1a53bddb0ff90ce"
I20260810 19:01:39.233181 20708 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c984e18c807b4f35a0d9138eec97f812" candidate_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00"
I20260810 19:01:39.233309 20708 raft_consensus.cc:3037] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:39.233304 20490 raft_consensus.cc:3037] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:01:39.234336 20490 raft_consensus.cc:2454] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ea371eba4948fe8503a51fa8dfec4a in term 1.
I20260810 19:01:39.234611 20708 raft_consensus.cc:2454] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c7ea371eba4948fe8503a51fa8dfec4a in term 1.
I20260810 19:01:39.234679 20155 leader_election.cc:302] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0556cee02b34302b1a53bddb0ff90ce, c7ea371eba4948fe8503a51fa8dfec4a; no voters:
I20260810 19:01:39.235009 20822 raft_consensus.cc:2781] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:01:39.235237 20822 raft_consensus.cc:686] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 LEADER]: Becoming Leader. State: Replica: c7ea371eba4948fe8503a51fa8dfec4a, State: Running, Role: LEADER
I20260810 19:01:39.235576 20822 consensus_queue.cc:237] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:39.238863 20015 catalog_manager.cc:5495] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a reported cstate change: term changed from 0 to 1, leader changed from <none> to c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129). New cstate: current_term: 1 leader_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } health_report { overall_health: UNKNOWN } } }
W20260810 19:01:39.249917 20571 tablet.cc:2369] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:01:39.269333 20346 tablet.cc:2369] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:01:39.302943 20801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.299607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41962 (local address 127.27.147.129:44319)
0810 19:01:39.299763 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:39.299767 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:39.299781 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:01:39.299884 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:39.299886 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:39.299945 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:39.300039 (+ 94us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:01:39.300265 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.300768 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.301601 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:39.301820 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:39.301906 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:39.302171 (+ 265us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:01:39.302565 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:01:39.302667 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:39.302713 (+ 46us) server_negotiation.cc:299] Negotiation successful
0810 19:01:39.302765 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 19:01:39.308794 20490 raft_consensus.cc:1261] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Refusing update from remote peer c7ea371eba4948fe8503a51fa8dfec4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:39.308795 20708 raft_consensus.cc:1261] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 FOLLOWER]: Refusing update from remote peer c7ea371eba4948fe8503a51fa8dfec4a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:01:39.309520 20822 consensus_queue.cc:1035] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:39.309729 20826 consensus_queue.cc:1035] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [LEADER]: Connected to new peer: Peer: permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:01:39.320689 20834 mvcc.cc:205] Tried to move back new op lower bound from 7317047293161177088 to 7317047292871667712. Current Snapshot: MvccSnapshot[applied={T|T < 7317047293161177088 or (T in {7317047293161177088})}]
I20260810 19:01:39.325088 20015 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:39.325244 20015 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48616:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:01:39.327418 20015 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Sending DeleteTablet for 3 replicas of tablet c984e18c807b4f35a0d9138eec97f812
I20260810 19:01:39.328163 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 20125
I20260810 19:01:39.328127 20458 tablet_service.cc:1514] Processing DeleteTablet for tablet c984e18c807b4f35a0d9138eec97f812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:39 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:34416
I20260810 19:01:39.328202 20676 tablet_service.cc:1514] Processing DeleteTablet for tablet c984e18c807b4f35a0d9138eec97f812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:39 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:50766
I20260810 19:01:39.336575 20845 tablet_replica.cc:330] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: stopping tablet replica
I20260810 19:01:39.336922 20845 raft_consensus.cc:2229] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:39.337236 20845 raft_consensus.cc:2258] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:39.339640 20845 ts_tablet_manager.cc:1905] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:39.341017 20844 tablet_replica.cc:330] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: stopping tablet replica
I20260810 19:01:39.341329 20844 raft_consensus.cc:2229] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:39.341638 20844 raft_consensus.cc:2258] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:39.342407 20845 ts_tablet_manager.cc:1918] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:39.342543 20845 log.cc:1196] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/c984e18c807b4f35a0d9138eec97f812
I20260810 19:01:39.342917 20845 ts_tablet_manager.cc:1939] T c984e18c807b4f35a0d9138eec97f812 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Deleting consensus metadata
I20260810 19:01:39.343041 20844 ts_tablet_manager.cc:1905] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:39.344028 19997 catalog_manager.cc:4841] TS a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): tablet c984e18c807b4f35a0d9138eec97f812 (table test-table [id=ce6c20d7094649dfa93d9a779b688f61]) successfully deleted
I20260810 19:01:39.346474 20844 ts_tablet_manager.cc:1918] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:39.346582 20844 log.cc:1196] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/c984e18c807b4f35a0d9138eec97f812
I20260810 19:01:39.346931 20844 ts_tablet_manager.cc:1939] T c984e18c807b4f35a0d9138eec97f812 P c0556cee02b34302b1a53bddb0ff90ce: Deleting consensus metadata
I20260810 19:01:39.348090 19995 catalog_manager.cc:4841] TS c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): tablet c984e18c807b4f35a0d9138eec97f812 (table test-table [id=ce6c20d7094649dfa93d9a779b688f61]) successfully deleted
W20260810 19:01:39.349399 19997 connection.cc:535] client connection to 127.27.147.129:44319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 19:01:39.349535 19997 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 19:01:39.349738 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 20352
I20260810 19:01:39.351722 20805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:39.350336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44319 (local address 127.0.0.1:41974)
0810 19:01:39.351515 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:39.351610 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1065}
W20260810 19:01:39.351923 19997 catalog_manager.cc:4584] TS c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129:44319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c984e18c807b4f35a0d9138eec97f812: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44319: connect: Connection refused (error 111)
I20260810 19:01:39.364472 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 20578
I20260810 19:01:39.378274 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 19971
I20260810 19:01:59.392527 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38471
--webserver_interface=127.27.147.190
--webserver_port=40017
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:01:59.510403 20848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:59.510645 20848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:59.510684 20848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:59.514595 20848 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:01:59.514642 20848 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:01:59.514657 20848 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:01:59.514671 20848 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:01:59.514689 20848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:59.514750 20848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:59.514780 20848 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:01:59.514849 20848 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:01:59.519953 20848 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:59.520918 20848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:59.521937 20848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:01:59.536250 20848 server_base.cc:1034] running on GCE node
W20260810 19:01:59.536194 20864 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:59.536193 20867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:59.536194 20861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:59.537073 20848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:59.538199 20848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:59.539391 20848 hybrid_clock.cc:648] HybridClock initialized: now 1786388519539356 us; error 46 us; skew 500 ppm
Aug 10 19:01:59 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388519, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:59.547493 20848 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:01:59.548388 20848 webserver.cc:460] Webserver started at http://127.27.147.190:40017/ using document root <none> and password file <none>
I20260810 19:01:59.548938 20848 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:59.549009 20848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:59.553068 20848 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 19:01:59.555418 20881 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:59.556759 20848 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:01:59.556875 20848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "94013a28caaa4bae854ed59213d2cdbf"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "c450e2634304f39b044c3dd856bf07db"
server_key_iv: "f087c98d5d4d07207e38e4dcfd39c88d"
server_key_version: "encryptionkey@0"
I20260810 19:01:59.557464 20848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:59.576076 20848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:59.579236 20848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:59.579401 20848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:59.586884 20848 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38471
I20260810 19:01:59.587226 20968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38471 every 8 connection(s)
I20260810 19:01:59.587992 20848 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:01:59.590900 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 20848
I20260810 19:01:59.591295 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:44319
--local_ip_for_outbound_sockets=127.27.147.129
--tserver_master_addrs=127.27.147.190:38471
--webserver_port=34613
--webserver_interface=127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:01:59.593012 20970 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:01:59.594856 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Bootstrap starting.
I20260810 19:01:59.614449 20970 log.cc:824] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:59.623966 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:01:59.624366 20970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Bootstrap complete.
I20260810 19:01:59.628881 20970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:59.629325 20970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94013a28caaa4bae854ed59213d2cdbf, State: Initialized, Role: FOLLOWER
I20260810 19:01:59.629890 20970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:59.630044 20970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:01:59.630143 20970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:01:59.630266 20970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 1 FOLLOWER]: Advancing to term 2
I20260810 19:01:59.632354 20970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:59.632714 20970 leader_election.cc:302] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94013a28caaa4bae854ed59213d2cdbf; no voters:
I20260810 19:01:59.633074 20970 leader_election.cc:288] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 19:01:59.633209 20975 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 FOLLOWER]: Leader election won for term 2
I20260810 19:01:59.633462 20975 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 LEADER]: Becoming Leader. State: Replica: 94013a28caaa4bae854ed59213d2cdbf, State: Running, Role: LEADER
I20260810 19:01:59.633872 20975 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:01:59.634395 20970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:01:59.634922 20975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94013a28caaa4bae854ed59213d2cdbf. Latest consensus state: current_term: 2 leader_uuid: "94013a28caaa4bae854ed59213d2cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } } }
I20260810 19:01:59.635128 20975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:59.635406 20977 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "94013a28caaa4bae854ed59213d2cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } } }
I20260810 19:01:59.635582 20980 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:01:59.635607 20977 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:01:59.640805 20980 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:01:59.641621 20980 catalog_manager.cc:1259] Loaded cluster ID: 653b1fa2af4e4881bb3e9d459f47b404
I20260810 19:01:59.641681 20980 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:01:59.644114 20980 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:01:59.644855 20980 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Loaded TSK: 0
I20260810 19:01:59.662189 20980 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Generated new TSK 1
I20260810 19:01:59.662501 20980 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:01:59.716732 20973 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:59.716972 20973 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:59.717015 20973 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:59.721019 20973 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:59.721072 20973 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:59.721159 20973 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:01:59.726269 20973 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:44319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=34613
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20973
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:59.727319 20973 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:59.728308 20973 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:59.739164 21013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:59.739112 21007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:59.739111 21006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:59.739842 20973 server_base.cc:1034] running on GCE node
I20260810 19:01:59.740214 20973 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:59.740819 20973 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:59.742005 20973 hybrid_clock.cc:648] HybridClock initialized: now 1786388519741986 us; error 34 us; skew 500 ppm
Aug 10 19:01:59 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388519, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:59.744760 20973 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:01:59.745703 20973 webserver.cc:460] Webserver started at http://127.27.147.129:34613/ using document root <none> and password file <none>
I20260810 19:01:59.746251 20973 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:59.746325 20973 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:59.750267 20973 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:01:59.752607 21021 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:59.753933 20973 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:01:59.754069 20973 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "c7ea371eba4948fe8503a51fa8dfec4a"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "2aff3f54c8833f0c3d83abb1398fc430"
server_key_iv: "44c9fb723b5f45789bc290e5e072c3fa"
server_key_version: "encryptionkey@0"
I20260810 19:01:59.754616 20973 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:01:59.798738 20973 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:01:59.802148 20973 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:01:59.802364 20973 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:01:59.803021 20973 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:01:59.804145 21035 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 19:01:59.808199 20973 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 19:01:59.808296 20973 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.004s user 0.000s sys 0.000s
I20260810 19:01:59.808383 20973 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 19:01:59.810081 20973 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 19:01:59.810149 20973 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.000s sys 0.000s
I20260810 19:01:59.810413 21035 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap starting.
I20260810 19:01:59.820947 20973 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:44319
I20260810 19:01:59.821043 21215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:44319 every 8 connection(s)
I20260810 19:01:59.821875 20973 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:01:59.830076 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 20973
I20260810 19:01:59.830322 21035 log.cc:824] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:01:59.830578 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:44475
--local_ip_for_outbound_sockets=127.27.147.130
--tserver_master_addrs=127.27.147.190:38471
--webserver_port=43051
--webserver_interface=127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:01:59 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388519, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:01:59.838112 21035 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:01:59.838501 21035 tablet_bootstrap.cc:492] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap complete.
I20260810 19:01:59.839952 21035 ts_tablet_manager.cc:1397] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.012s
I20260810 19:01:59.841568 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:59.824095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:40887 (local address 127.27.147.190:38471)
0810 19:01:59.824568 (+ 473us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:59.824587 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:59.824704 (+ 117us) server_negotiation.cc:407] Connection header received
0810 19:01:59.826195 (+ 1491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:59.826207 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:59.826519 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:59.826777 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:59.828346 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.829320 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.830970 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.831682 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.831858 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:59.835078 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:59.835087 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:59.835106 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:59.835147 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:59.837696 (+ 2549us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:59.838347 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:59.838356 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:59.838366 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:59.838539 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:59.839077 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:59.839084 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:59.839088 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:59.839556 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:59.839696 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:59.840419 (+ 723us) server_negotiation.cc:299] Negotiation successful
0810 19:01:59.840716 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":121,"threads_started":1}
I20260810 19:01:59.841686 21220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:59.823941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:40887)
0810 19:01:59.824526 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:59.824582 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:59.825776 (+ 1194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:59.826975 (+ 1199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:59.826984 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:59.827444 (+ 460us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:59.828151 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:59.828181 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.829477 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.829481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:59.830791 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:59.830803 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.831866 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.831873 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:59.832119 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:59.832846 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:59.832873 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:59.834874 (+ 2001us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:59.837834 (+ 2960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:59.837847 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:59.837858 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:59.838191 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:59.838690 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:59.838695 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:59.838700 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:59.838947 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:59.839693 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:59.839703 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:59.840042 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 19:01:59.840275 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 19:01:59.840647 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":165,"threads_started":1}
I20260810 19:01:59.842913 21035 raft_consensus.cc:348] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:59.843452 21035 raft_consensus.cc:729] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ea371eba4948fe8503a51fa8dfec4a, State: Initialized, Role: FOLLOWER
I20260810 19:01:59.843933 21035 consensus_queue.cc:260] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } }
I20260810 19:01:59.844358 21216 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38471
I20260810 19:01:59.844561 21216 heartbeater.cc:464] Registering TS with master...
I20260810 19:01:59.844926 21035 ts_tablet_manager.cc:1428] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:01:59.845083 21216 heartbeater.cc:511] Master 127.27.147.190:38471 requested a full tablet report, sending...
I20260810 19:01:59.847424 20908 ts_manager.cc:194] Registered new tserver with Master: c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129:44319)
I20260810 19:01:59.850625 20908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:40887
I20260810 19:01:59.853339 21216 heartbeater.cc:503] Master 127.27.147.190:38471 was elected leader, sending a full tablet report...
Aug 10 19:01:59 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388519, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.129@KRBTEST.COM
I20260810 19:01:59.861058 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:59.850129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44319 (local address 127.0.0.1:57496)
0810 19:01:59.850747 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 19:01:59.850787 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:01:59.850991 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:01:59.851691 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:01:59.851702 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:01:59.852002 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:01:59.852100 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:59.852106 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.853053 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.853058 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:59.853783 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:01:59.853799 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.854554 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.854559 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:01:59.854686 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:59.855243 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:01:59.855259 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:01:59.856848 (+ 1589us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:01:59.859032 (+ 2184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:59.859054 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:59.859060 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:59.859312 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:59.859620 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:01:59.859623 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:01:59.859624 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:01:59.859700 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:01:59.860212 (+ 512us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:01:59.860217 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:01:59.860316 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 19:01:59.860672 (+ 356us) client_negotiation.cc:240] Negotiation successful
0810 19:01:59.860902 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":123,"threads_started":1}
I20260810 19:01:59.861152 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:01:59.850167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.27.147.129:44319)
0810 19:01:59.850485 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 19:01:59.850496 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:01:59.850894 (+ 398us) server_negotiation.cc:407] Connection header received
0810 19:01:59.851153 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:01:59.851158 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:01:59.851356 (+ 198us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:01:59.851489 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:01:59.852218 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.852941 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.853898 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:01:59.854395 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:01:59.854512 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:01:59.857003 (+ 2491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:01:59.857012 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:01:59.857021 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:01:59.857042 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:01:59.858933 (+ 1891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:59.859424 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:59.859430 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:59.859432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:59.859526 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:01:59.859783 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:01:59.859786 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:01:59.859788 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:01:59.860065 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:01:59.860203 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 19:01:59.860808 (+ 605us) server_negotiation.cc:299] Negotiation successful
0810 19:01:59.860973 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":244,"thread_start_us":89,"threads_started":1}
I20260810 19:01:59.862262 21103 tablet_service.cc:1514] Processing DeleteTablet for tablet c984e18c807b4f35a0d9138eec97f812 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:01:39 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:57496
I20260810 19:01:59.863262 21229 tablet_replica.cc:330] stopping tablet replica
I20260810 19:01:59.863554 21229 raft_consensus.cc:2229] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:01:59.863797 21229 raft_consensus.cc:2258] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:01:59.864692 21229 ts_tablet_manager.cc:1905] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:01:59.868440 21229 ts_tablet_manager.cc:1918] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:01:59.868548 21229 log.cc:1196] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal/wals/c984e18c807b4f35a0d9138eec97f812
I20260810 19:01:59.868826 21229 ts_tablet_manager.cc:1939] T c984e18c807b4f35a0d9138eec97f812 P c7ea371eba4948fe8503a51fa8dfec4a: Deleting consensus metadata
I20260810 19:01:59.870402 20889 catalog_manager.cc:4841] TS c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129:44319): tablet c984e18c807b4f35a0d9138eec97f812 (table test-table [id=ce6c20d7094649dfa93d9a779b688f61]) successfully deleted
W20260810 19:01:59.951687 21224 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:01:59.951917 21224 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:01:59.951958 21224 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:01:59.955840 21224 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:01:59.955891 21224 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:01:59.955945 21224 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:01:59.961043 21224 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:44475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=43051
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21224
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:01:59.962019 21224 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:01:59.962972 21224 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:01:59.972218 21240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:59.972254 21243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:01:59.972250 21241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:01:59.972435 21224 server_base.cc:1034] running on GCE node
I20260810 19:01:59.972918 21224 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:01:59.973534 21224 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:01:59.974712 21224 hybrid_clock.cc:648] HybridClock initialized: now 1786388519974697 us; error 25 us; skew 500 ppm
Aug 10 19:01:59 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388519, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:01:59.977288 21224 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:01:59.978089 21224 webserver.cc:460] Webserver started at http://127.27.147.130:43051/ using document root <none> and password file <none>
I20260810 19:01:59.978685 21224 fs_manager.cc:359] Metadata directory not provided
I20260810 19:01:59.978790 21224 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:01:59.982429 21224 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260810 19:01:59.984587 21258 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:01:59.985769 21224 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260810 19:01:59.985911 21224 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "c0556cee02b34302b1a53bddb0ff90ce"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "faa86205bf82e41fd9ae6b98619af665"
server_key_iv: "80b3ff7bf30066fe3605c9659c30dbb4"
server_key_version: "encryptionkey@0"
I20260810 19:01:59.986413 21224 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:00.006654 21224 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:00.009760 21224 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:00.010062 21224 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:00.010660 21224 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:00.011548 21224 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:00.011617 21224 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:00.011685 21224 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:00.011731 21224 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:00.020720 21224 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:44475
I20260810 19:02:00.020761 21442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:44475 every 8 connection(s)
I20260810 19:02:00.021687 21224 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:02:00.029229 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 21224
I20260810 19:02:00.029651 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:35945
--local_ip_for_outbound_sockets=127.27.147.131
--tserver_master_addrs=127.27.147.190:38471
--webserver_port=40217
--webserver_interface=127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 19:02:00 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388519, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:00.035362 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.023380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:37147 (local address 127.27.147.190:38471)
0810 19:02:00.023528 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.023532 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.024606 (+ 1074us) server_negotiation.cc:407] Connection header received
0810 19:02:00.025525 (+ 919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.025532 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.025596 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.025699 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:00.026852 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.027405 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.028551 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.028806 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.028919 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.031553 (+ 2634us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:00.031556 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:00.031559 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:00.031582 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:00.033058 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.033644 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.033651 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.033653 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.033748 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.034113 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.034118 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.034120 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.034322 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:00.034431 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.034969 (+ 538us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.035132 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:02:00.036127 21445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.023765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:37147)
0810 19:02:00.024435 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:00.024475 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:00.025310 (+ 835us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:00.025878 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:00.025885 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:00.026218 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:00.026671 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.026684 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.027535 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.027543 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.028422 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.028432 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.028958 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.028962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.029148 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.029784 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:00.029821 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:00.031327 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:00.033178 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:00.033183 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:00.033195 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:00.033519 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:00.033843 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:00.033847 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:00.033849 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:00.034007 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:00.034431 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:00.034439 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:00.034669 (+ 230us) client_negotiation.cc:769] Sending connection context
0810 19:02:00.034918 (+ 249us) client_negotiation.cc:240] Negotiation successful
0810 19:02:00.035238 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":488,"thread_start_us":124,"threads_started":1}
I20260810 19:02:00.037261 21443 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38471
I20260810 19:02:00.037503 21443 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:00.037968 21443 heartbeater.cc:511] Master 127.27.147.190:38471 requested a full tablet report, sending...
I20260810 19:02:00.039075 20908 ts_manager.cc:194] Registered new tserver with Master: c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475)
I20260810 19:02:00.039953 20908 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:37147
W20260810 19:02:00.144052 21447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:00.144284 21447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:00.144325 21447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:00.148155 21447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:00.148200 21447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:00.148254 21447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:00.153301 21447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:35945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=40217
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:00.154220 21447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:00.155216 21447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:00.165828 21458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:00.165840 21461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:00.165843 21463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:00.166138 21447 server_base.cc:1034] running on GCE node
I20260810 19:02:00.166633 21447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:00.167271 21447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:00.168444 21447 hybrid_clock.cc:648] HybridClock initialized: now 1786388520168426 us; error 27 us; skew 500 ppm
Aug 10 19:02:00 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388520, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:00.171267 21447 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:02:00.172142 21447 webserver.cc:460] Webserver started at http://127.27.147.131:40217/ using document root <none> and password file <none>
I20260810 19:02:00.172699 21447 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:00.172809 21447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:00.176646 21447 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:02:00.178637 21473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:00.179926 21447 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:00.180058 21447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "fe573f7086a74540be878a3e249ffc9e"
server_key_iv: "daf0d21456279f20a398ae58d2a7fdff"
server_key_version: "encryptionkey@0"
I20260810 19:02:00.180579 21447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:00.196265 21447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:00.199334 21447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:00.199527 21447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:00.200114 21447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:00.200915 21447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:00.200984 21447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:00.201054 21447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:00.201094 21447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:00.210464 21447 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:35945
I20260810 19:02:00.210500 21646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:35945 every 8 connection(s)
I20260810 19:02:00.211474 21447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:02:00.217424 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 21447
Aug 10 19:02:00 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388520, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:00.225843 21221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.213252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:55479 (local address 127.27.147.190:38471)
0810 19:02:00.213408 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.213412 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.214428 (+ 1016us) server_negotiation.cc:407] Connection header received
0810 19:02:00.215294 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.215297 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.215344 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.215434 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:00.216561 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.217062 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.218628 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.218993 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.219119 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.221948 (+ 2829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:00.221951 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:00.221953 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:00.221974 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:00.223664 (+ 1690us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.224231 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.224233 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.224235 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.224312 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.224698 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.224700 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.224702 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.224867 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:00.224944 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.225562 (+ 618us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.225705 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":70}
I20260810 19:02:00.226953 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.218107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46418 (local address 127.27.147.190:38471)
0810 19:02:00.218350 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.218355 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.218541 (+ 186us) server_negotiation.cc:407] Connection header received
0810 19:02:00.218672 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.218675 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.218714 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.218792 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:00.219804 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.220592 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.221743 (+ 1151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.222044 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.222120 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.223613 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:00.223616 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:00.223617 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:00.223639 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:00.225230 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.225690 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.225693 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.225695 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.225819 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.226135 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.226138 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.226140 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.226339 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:00.226536 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.226654 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.226764 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"spinlock_wait_cycles":1792,"thread_start_us":88,"threads_started":1}
I20260810 19:02:00.226949 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.213625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:55479)
0810 19:02:00.214243 (+ 618us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:00.214282 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:00.215107 (+ 825us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:00.215598 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:00.215604 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:00.215952 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:00.216430 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.216444 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.217208 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.217213 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.218479 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.218490 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.219147 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.219150 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.219288 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.219908 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:00.219928 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:00.221790 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:00.223771 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:00.223780 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:00.223794 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:00.224138 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:00.224408 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:00.224414 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:00.224417 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:00.224585 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:00.224976 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:00.224993 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:00.225297 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 19:02:00.225480 (+ 183us) client_negotiation.cc:240] Negotiation successful
0810 19:02:00.225765 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":108,"threads_started":1}
I20260810 19:02:00.228019 21647 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38471
I20260810 19:02:00.228251 21647 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:00.228802 21647 heartbeater.cc:511] Master 127.27.147.190:38471 requested a full tablet report, sending...
I20260810 19:02:00.229753 20904 ts_manager.cc:194] Registered new tserver with Master: a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
I20260810 19:02:00.230350 20904 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:55479
I20260810 19:02:00.240168 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:02:00.243641 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.240706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46428 (local address 127.27.147.190:38471)
0810 19:02:00.240821 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.240824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.240969 (+ 145us) server_negotiation.cc:407] Connection header received
0810 19:02:00.241087 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.241089 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.241167 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.241246 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:00.241420 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.241877 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.242661 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.242930 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.243009 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.243271 (+ 262us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:00.243379 (+ 108us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 19:02:00.243560 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46428: token expired
Metrics: {"server-negotiator.queue_time_us":42}
W20260810 19:02:00.243716 21650 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46428: token expired
W20260810 19:02:00.244069 20787 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.27.147.190:38471)
I20260810 19:02:00.250432 21650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.244449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46432 (local address 127.27.147.190:38471)
0810 19:02:00.244550 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.244553 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.244565 (+ 12us) server_negotiation.cc:407] Connection header received
0810 19:02:00.244594 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.244596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.244625 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.244693 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:00.245225 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.245678 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.246381 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.246574 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.246647 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.247896 (+ 1249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:00.247899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:00.247901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:00.247922 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:00.249124 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.249490 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.249493 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.249494 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.249568 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:00.249786 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:00.249788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:00.249789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:00.249935 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:00.250031 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.250223 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.250336 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260810 19:02:00.252840 20904 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:02:00.256003 20904 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:02:00.263814 21103 tablet_service.cc:1467] Processing CreateTablet for tablet ee233c92d47f49b1a992e224a021a373 (DEFAULT_TABLE table=test-table [id=dedfd14823814b50a3076490fe8855d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:00.264473 21103 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee233c92d47f49b1a992e224a021a373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:00.267639 21226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.262306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44475 (local address 127.0.0.1:48450)
0810 19:02:00.262476 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:00.262489 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:00.262562 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:00.263442 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:00.263445 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:00.263466 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:00.263561 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.263567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.264885 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.264888 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.266055 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.266065 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.267169 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.267176 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.267270 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.267315 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 19:02:00.267415 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:02:00.267498 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 19:02:00.267709 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.262511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48450 (local address 127.27.147.130:44475)
0810 19:02:00.262895 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.262918 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.262946 (+ 28us) server_negotiation.cc:407] Connection header received
0810 19:02:00.263030 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.263036 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.263269 (+ 233us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.263453 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:00.263670 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.264741 (+ 1071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.266179 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.267057 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.267186 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.267325 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.267448 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.267551 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":171,"threads_started":1}
I20260810 19:02:00.267865 21035 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap starting.
I20260810 19:02:00.268409 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.262782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:35945 (local address 127.0.0.1:60746)
0810 19:02:00.263055 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:00.263069 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:00.263151 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:00.263939 (+ 788us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:00.263942 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:00.263959 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:00.264053 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.264059 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.265260 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.265264 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.266363 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.266377 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.267658 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.267662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.268083 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.268115 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 19:02:00.268198 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:02:00.268275 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":149,"threads_started":1}
I20260810 19:02:00.268625 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.262967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60746 (local address 127.27.147.131:35945)
0810 19:02:00.263536 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.263549 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.263571 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:02:00.263645 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.263650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.263810 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.263963 (+ 153us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:00.264230 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.265123 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.266516 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.267360 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.267479 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.267591 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.268266 (+ 675us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.268363 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":476,"thread_start_us":150,"threads_started":1}
I20260810 19:02:00.268818 21035 tablet_bootstrap.cc:654] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:00.270298 21035 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: No bootstrap required, opened a new log
I20260810 19:02:00.270392 21035 ts_tablet_manager.cc:1397] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260810 19:02:00.270871 21035 raft_consensus.cc:348] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.271005 21035 raft_consensus.cc:374] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:00.271051 21035 raft_consensus.cc:729] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ea371eba4948fe8503a51fa8dfec4a, State: Initialized, Role: FOLLOWER
I20260810 19:02:00.271211 21035 consensus_queue.cc:260] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.271565 21035 ts_tablet_manager.cc:1428] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260810 19:02:00.271641 21539 tablet_service.cc:1467] Processing CreateTablet for tablet ee233c92d47f49b1a992e224a021a373 (DEFAULT_TABLE table=test-table [id=dedfd14823814b50a3076490fe8855d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:00.272722 21539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee233c92d47f49b1a992e224a021a373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:00.272294 21337 tablet_service.cc:1467] Processing CreateTablet for tablet ee233c92d47f49b1a992e224a021a373 (DEFAULT_TABLE table=test-table [id=dedfd14823814b50a3076490fe8855d9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:00.273342 21337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee233c92d47f49b1a992e224a021a373. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:00.279501 21666 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Bootstrap starting.
I20260810 19:02:00.279779 21667 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Bootstrap starting.
I20260810 19:02:00.281968 21667 tablet_bootstrap.cc:654] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:00.281973 21666 tablet_bootstrap.cc:654] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:00.282676 21667 log.cc:824] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:00.282676 21666 log.cc:824] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:00.284814 21666 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: No bootstrap required, opened a new log
I20260810 19:02:00.284821 21667 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: No bootstrap required, opened a new log
I20260810 19:02:00.285015 21666 ts_tablet_manager.cc:1397] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 19:02:00.285012 21667 ts_tablet_manager.cc:1397] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 19:02:00.287873 21666 raft_consensus.cc:348] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.287873 21667 raft_consensus.cc:348] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.288115 21666 raft_consensus.cc:374] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:00.288110 21667 raft_consensus.cc:374] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:00.288177 21666 raft_consensus.cc:729] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a88d6cfe7bbd4cf9858a973c3d7a9f00, State: Initialized, Role: FOLLOWER
I20260810 19:02:00.288187 21667 raft_consensus.cc:729] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0556cee02b34302b1a53bddb0ff90ce, State: Initialized, Role: FOLLOWER
I20260810 19:02:00.288591 21667 consensus_queue.cc:260] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.288591 21666 consensus_queue.cc:260] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.289214 21443 heartbeater.cc:503] Master 127.27.147.190:38471 was elected leader, sending a full tablet report...
I20260810 19:02:00.289548 21667 ts_tablet_manager.cc:1428] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Time spent starting tablet: real 0.004s user 0.005s sys 0.000s
I20260810 19:02:00.289614 21666 ts_tablet_manager.cc:1428] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:02:00.290078 21647 heartbeater.cc:503] Master 127.27.147.190:38471 was elected leader, sending a full tablet report...
W20260810 19:02:00.323048 21218 tablet.cc:2369] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:02:00.348958 21670 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:00.349179 21670 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.350481 21670 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129:44319)
I20260810 19:02:00.355497 21658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.350926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:38729 (local address 127.27.147.130:44475)
0810 19:02:00.351052 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.351057 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.351072 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:02:00.351138 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.351141 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.351187 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.351256 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:00.351634 (+ 378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.352427 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.354081 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.354743 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.354923 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.354979 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.355295 (+ 316us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.355366 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:02:00.355497 21227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.350921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:52057 (local address 127.27.147.129:44319)
0810 19:02:00.351087 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.351092 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.351329 (+ 237us) server_negotiation.cc:407] Connection header received
0810 19:02:00.351491 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.351494 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.351549 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.351671 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:00.351913 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.352634 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.354081 (+ 1447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.354743 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.354923 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.355076 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.355278 (+ 202us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.355360 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 19:02:00.355564 21649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.350665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44475 (local address 127.27.147.131:38729)
0810 19:02:00.350839 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:00.350859 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:00.350957 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:00.351297 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:00.351301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:00.351335 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:00.351521 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.351526 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.352583 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.352587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.353915 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.353926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.354916 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.354919 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.355107 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.355167 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 19:02:00.355248 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:02:00.355400 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":111,"spinlock_wait_cycles":2944}
I20260810 19:02:00.355769 21674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.350810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44319 (local address 127.27.147.131:52057)
0810 19:02:00.351203 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:00.351213 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:00.351355 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:00.351693 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:00.351695 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:00.351710 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:00.351802 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.351806 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.352761 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.352764 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.353915 (+ 1151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:00.353926 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.354900 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.354904 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:00.355111 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.355165 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 19:02:00.355248 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 19:02:00.355407 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":103,"spinlock_wait_cycles":50304,"thread_start_us":124,"threads_started":1}
I20260810 19:02:00.356308 21376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee233c92d47f49b1a992e224a021a373" candidate_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0556cee02b34302b1a53bddb0ff90ce" is_pre_election: true
I20260810 19:02:00.356387 21137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee233c92d47f49b1a992e224a021a373" candidate_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" is_pre_election: true
I20260810 19:02:00.356675 21137 raft_consensus.cc:2454] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88d6cfe7bbd4cf9858a973c3d7a9f00 in term 0.
I20260810 19:02:00.356675 21376 raft_consensus.cc:2454] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a88d6cfe7bbd4cf9858a973c3d7a9f00 in term 0.
I20260810 19:02:00.357210 21480 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce, c7ea371eba4948fe8503a51fa8dfec4a; no voters:
I20260810 19:02:00.357501 21670 raft_consensus.cc:2781] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:02:00.357586 21670 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:02:00.357618 21670 raft_consensus.cc:3037] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:00.358762 21670 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.359166 21670 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [CANDIDATE]: Term 1 election: Requested vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129:44319)
I20260810 19:02:00.359575 21137 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee233c92d47f49b1a992e224a021a373" candidate_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7ea371eba4948fe8503a51fa8dfec4a"
I20260810 19:02:00.359692 21137 raft_consensus.cc:3037] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:00.360798 21137 raft_consensus.cc:2454] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88d6cfe7bbd4cf9858a973c3d7a9f00 in term 1.
I20260810 19:02:00.361088 21480 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c7ea371eba4948fe8503a51fa8dfec4a; no voters:
I20260810 19:02:00.361079 21376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee233c92d47f49b1a992e224a021a373" candidate_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0556cee02b34302b1a53bddb0ff90ce"
I20260810 19:02:00.361202 21376 raft_consensus.cc:3037] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:00.361264 21670 raft_consensus.cc:2781] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:00.361496 21670 raft_consensus.cc:686] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 LEADER]: Becoming Leader. State: Replica: a88d6cfe7bbd4cf9858a973c3d7a9f00, State: Running, Role: LEADER
I20260810 19:02:00.361876 21670 consensus_queue.cc:237] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:00.362581 21376 raft_consensus.cc:2454] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a88d6cfe7bbd4cf9858a973c3d7a9f00 in term 1.
I20260810 19:02:00.365150 20904 catalog_manager.cc:5495] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 reported cstate change: term changed from 0 to 1, leader changed from <none> to a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131). New cstate: current_term: 1 leader_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } health_report { overall_health: UNKNOWN } } }
I20260810 19:02:00.394559 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.391471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60760 (local address 127.27.147.131:35945)
0810 19:02:00.391595 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:00.391599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:00.391629 (+ 30us) server_negotiation.cc:407] Connection header received
0810 19:02:00.391777 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:00.391780 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:00.391832 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:00.391919 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:00.392125 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.392620 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.393375 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:00.393577 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:00.393650 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:00.393895 (+ 245us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:00.394269 (+ 374us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:02:00.394354 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:00.394410 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 19:02:00.394457 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:02:00.400843 21376 raft_consensus.cc:1261] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Refusing update from remote peer a88d6cfe7bbd4cf9858a973c3d7a9f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:00.400853 21137 raft_consensus.cc:1261] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Refusing update from remote peer a88d6cfe7bbd4cf9858a973c3d7a9f00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:00.401574 21670 consensus_queue.cc:1035] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:00.401808 21676 consensus_queue.cc:1035] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:00.409576 21679 mvcc.cc:205] Tried to move back new op lower bound from 7317047379553284096 to 7317047379405672448. Current Snapshot: MvccSnapshot[applied={T|T < 7317047379553284096}]
I20260810 19:02:00.411234 21681 mvcc.cc:205] Tried to move back new op lower bound from 7317047379553284096 to 7317047379405672448. Current Snapshot: MvccSnapshot[applied={T|T < 7317047379553284096}]
I20260810 19:02:00.419144 20904 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:00.419281 20904 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46432:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:00.421437 20904 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Sending DeleteTablet for 3 replicas of tablet ee233c92d47f49b1a992e224a021a373
I20260810 19:02:00.422093 21337 tablet_service.cc:1514] Processing DeleteTablet for tablet ee233c92d47f49b1a992e224a021a373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:00 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:48450
I20260810 19:02:00.422220 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 20973
I20260810 19:02:00.422235 21539 tablet_service.cc:1514] Processing DeleteTablet for tablet ee233c92d47f49b1a992e224a021a373 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:00 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:60746
I20260810 19:02:00.422902 21701 tablet_replica.cc:330] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: stopping tablet replica
I20260810 19:02:00.423270 21701 raft_consensus.cc:2229] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:00.423630 21701 raft_consensus.cc:2258] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:00.426208 21701 ts_tablet_manager.cc:1905] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:00.429121 21701 ts_tablet_manager.cc:1918] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:00.429227 21701 log.cc:1196] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/ee233c92d47f49b1a992e224a021a373
I20260810 19:02:00.429534 21702 tablet_replica.cc:330] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: stopping tablet replica
I20260810 19:02:00.429585 21701 ts_tablet_manager.cc:1939] T ee233c92d47f49b1a992e224a021a373 P c0556cee02b34302b1a53bddb0ff90ce: Deleting consensus metadata
I20260810 19:02:00.429996 21702 raft_consensus.cc:2229] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:02:00.430459 21702 raft_consensus.cc:2258] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:00.431787 20885 catalog_manager.cc:4841] TS c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): tablet ee233c92d47f49b1a992e224a021a373 (table test-table [id=dedfd14823814b50a3076490fe8855d9]) successfully deleted
I20260810 19:02:00.435813 21702 ts_tablet_manager.cc:1905] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 19:02:00.436257 20889 connection.cc:535] client connection to 127.27.147.129:44319 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260810 19:02:00.436432 20889 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 19:02:00.437302 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 21224
I20260810 19:02:00.438886 21702 ts_tablet_manager.cc:1918] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:00.439015 21702 log.cc:1196] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/ee233c92d47f49b1a992e224a021a373
I20260810 19:02:00.439459 21702 ts_tablet_manager.cc:1939] T ee233c92d47f49b1a992e224a021a373 P a88d6cfe7bbd4cf9858a973c3d7a9f00: Deleting consensus metadata
I20260810 19:02:00.442075 20889 catalog_manager.cc:4841] TS a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): tablet ee233c92d47f49b1a992e224a021a373 (table test-table [id=dedfd14823814b50a3076490fe8855d9]) successfully deleted
I20260810 19:02:00.443428 21659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:00.443129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:44319 (local address 127.0.0.1:57500)
0810 19:02:00.443266 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:00.443341 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44319: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260810 19:02:00.443934 20889 catalog_manager.cc:4584] TS c7ea371eba4948fe8503a51fa8dfec4a (127.27.147.129:44319): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee233c92d47f49b1a992e224a021a373: Network error: Client connection negotiation failed: client connection to 127.27.147.129:44319: connect: Connection refused (error 111)
I20260810 19:02:00.448382 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 21447
I20260810 19:02:00.462388 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 20848
I20260810 19:02:20.482071 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38471
--webserver_interface=127.27.147.190
--webserver_port=40017
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38471
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:20.596992 21708 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:20.597224 21708 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:20.597262 21708 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:20.601377 21708 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 19:02:20.601423 21708 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 19:02:20.601439 21708 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 19:02:20.601452 21708 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:20.601500 21708 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:20.601536 21708 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:20.601555 21708 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:20.601596 21708 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:20.606624 21708 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38471
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=40017
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21708
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:20.607575 21708 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:20.608521 21708 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:20.617578 21720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:20.617640 21715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:20.617669 21717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:20.617805 21708 server_base.cc:1034] running on GCE node
I20260810 19:02:20.618396 21708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:20.619441 21708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:20.620636 21708 hybrid_clock.cc:648] HybridClock initialized: now 1786388540620617 us; error 30 us; skew 500 ppm
Aug 10 19:02:20 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:20.628648 21708 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:02:20.629556 21708 webserver.cc:460] Webserver started at http://127.27.147.190:40017/ using document root <none> and password file <none>
I20260810 19:02:20.630118 21708 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:20.630211 21708 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:20.633950 21708 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:20.636087 21731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:20.637243 21708 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:02:20.637396 21708 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "94013a28caaa4bae854ed59213d2cdbf"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "c450e2634304f39b044c3dd856bf07db"
server_key_iv: "f087c98d5d4d07207e38e4dcfd39c88d"
server_key_version: "encryptionkey@0"
I20260810 19:02:20.637919 21708 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:20.664266 21708 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:20.667089 21708 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:20.667296 21708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:20.674468 21708 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38471
I20260810 19:02:20.674752 21808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38471 every 8 connection(s)
I20260810 19:02:20.675550 21708 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:20.679773 21809 sys_catalog.cc:263] Verifying existing consensus state
I20260810 19:02:20.680225 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 21708
I20260810 19:02:20.680712 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:44319
--local_ip_for_outbound_sockets=127.27.147.129
--tserver_master_addrs=127.27.147.190:38471
--webserver_port=34613
--webserver_interface=127.27.147.129
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:02:20.681945 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Bootstrap starting.
I20260810 19:02:20.700676 21809 log.cc:824] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:20.711755 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:02:20.712127 21809 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Bootstrap complete.
I20260810 19:02:20.716241 21809 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:02:20.716595 21809 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94013a28caaa4bae854ed59213d2cdbf, State: Initialized, Role: FOLLOWER
I20260810 19:02:20.717057 21809 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:02:20.717165 21809 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:20.717254 21809 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:20.717363 21809 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 2 FOLLOWER]: Advancing to term 3
I20260810 19:02:20.719178 21809 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:02:20.719498 21809 leader_election.cc:302] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94013a28caaa4bae854ed59213d2cdbf; no voters:
I20260810 19:02:20.719816 21809 leader_election.cc:288] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 19:02:20.719897 21820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 3 FOLLOWER]: Leader election won for term 3
I20260810 19:02:20.720088 21820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [term 3 LEADER]: Becoming Leader. State: Replica: 94013a28caaa4bae854ed59213d2cdbf, State: Running, Role: LEADER
I20260810 19:02:20.720368 21820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } }
I20260810 19:02:20.721016 21809 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:20.721282 21821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "94013a28caaa4bae854ed59213d2cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } } }
I20260810 19:02:20.722023 21821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:20.721905 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94013a28caaa4bae854ed59213d2cdbf. Latest consensus state: current_term: 3 leader_uuid: "94013a28caaa4bae854ed59213d2cdbf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94013a28caaa4bae854ed59213d2cdbf" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38471 } } }
I20260810 19:02:20.722350 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:20.722643 21837 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:20.728035 21837 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:20.728749 21837 catalog_manager.cc:1259] Loaded cluster ID: 653b1fa2af4e4881bb3e9d459f47b404
I20260810 19:02:20.728813 21837 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:20.731210 21837 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:20.731915 21837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Loaded TSK: 0
I20260810 19:02:20.732005 21837 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 94013a28caaa4bae854ed59213d2cdbf: Loaded TSK: 1
F20260810 19:02:20.732707 21837 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f4def0b59ad google::LogMessage::Fail() at ??:0
@ 0x7f4def0b888a google::LogMessage::SendToLog() at ??:0
@ 0x7f4def0b54de google::LogMessage::Flush() at ??:0
@ 0x7f4def0b6a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f4df1d7918d kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f4df1d71f0d kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f4df9a1a7bb kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f4df99f499c kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f4df99f5178 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlvE3_clEv at ??:0
@ 0x7f4df9a36df8 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f4df9a5582b std::function<>::operator()() at ??:0
@ 0x7f4df99f4b4f _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_ at ??:0
@ 0x7f4df99f5f3e kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f4df99f2579 _ZZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvENKUlvE_clEv at ??:0
@ 0x7f4df9a36640 _ZNSt17_Function_handlerIFvvEZN4kudu6master14CatalogManager17ElectedAsLeaderCbEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4df13073d7 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f4df1307c83 _ZZN4kudu10ThreadPool12CreateThreadEvENKUlvE_clEv at ??:0
@ 0x7f4df13093f2 _ZNSt17_Function_handlerIFvvEZN4kudu10ThreadPool12CreateThreadEvEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x61d838 std::function<>::operator()() at ??:0
@ 0x7f4df12f8512 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f4df293c6db start_thread at ??:0
@ 0x7f4dedbb361f clone at ??:0
W20260810 19:02:20.822301 21815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:20.822517 21815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:20.822558 21815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:20.826603 21815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:20.826682 21815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:20.826798 21815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:02:20.831916 21815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:44319
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=34613
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:20.832912 21815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:20.833921 21815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:20.843598 21891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:20.843621 21893 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:20.843838 21815 server_base.cc:1034] running on GCE node
W20260810 19:02:20.843971 21899 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:20.844431 21815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:20.845196 21815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:20.846410 21815 hybrid_clock.cc:648] HybridClock initialized: now 1786388540846382 us; error 32 us; skew 500 ppm
Aug 10 19:02:20 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:20.849650 21815 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:02:20.850467 21815 webserver.cc:460] Webserver started at http://127.27.147.129:34613/ using document root <none> and password file <none>
I20260810 19:02:20.851070 21815 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:20.851136 21815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:20.854979 21815 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:02:20.857174 21907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:20.858496 21815 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:20.858630 21815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "c7ea371eba4948fe8503a51fa8dfec4a"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "2aff3f54c8833f0c3d83abb1398fc430"
server_key_iv: "44c9fb723b5f45789bc290e5e072c3fa"
server_key_version: "encryptionkey@0"
I20260810 19:02:20.859206 21815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:20.871189 21815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:20.874962 21815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:20.875166 21815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:20.875898 21815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:20.877257 21916 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260810 19:02:20.881548 21815 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 19:02:20.881628 21815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.001s sys 0.000s
I20260810 19:02:20.881695 21815 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 19:02:20.883989 21815 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 19:02:20.884056 21815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.002s user 0.001s sys 0.000s
I20260810 19:02:20.884474 21916 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap starting.
I20260810 19:02:20.895394 21815 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:44319
I20260810 19:02:20.896627 21815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:02:20.897570 22082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:44319 every 8 connection(s)
I20260810 19:02:20.898847 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 21815
I20260810 19:02:20.899425 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:44475
--local_ip_for_outbound_sockets=127.27.147.130
--tserver_master_addrs=127.27.147.190:38471
--webserver_port=43051
--webserver_interface=127.27.147.130
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:02:20.905628 21916 log.cc:824] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:20.912642 21916 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 19:02:20.913035 21916 tablet_bootstrap.cc:492] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Bootstrap complete.
I20260810 19:02:20.914234 21916 ts_tablet_manager.cc:1397] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent bootstrapping tablet: real 0.030s user 0.022s sys 0.004s
I20260810 19:02:20.916646 21916 raft_consensus.cc:348] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:20.917073 21916 raft_consensus.cc:729] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7ea371eba4948fe8503a51fa8dfec4a, State: Initialized, Role: FOLLOWER
I20260810 19:02:20.917555 21916 consensus_queue.cc:260] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:20.918563 21916 ts_tablet_manager.cc:1428] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a: Time spent starting tablet: real 0.004s user 0.007s sys 0.000s
W20260810 19:02:20.993286 22092 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:02:20.899120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:47847)
0810 19:02:20.900707 (+ 1587us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:20.900766 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:20.901811 (+ 1045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:20.992305 (+ 90494us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":1348,"thread_start_us":147,"threads_started":1}
W20260810 19:02:20.993909 22083 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38471 (0 consecutive failures): Network error: Failed to ping master at 127.27.147.190:38471: Client connection negotiation failed: client connection to 127.27.147.190:38471: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 19:02:20.994694 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:20.994399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:45419)
0810 19:02:20.994557 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:20.994631 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 19:02:20.995436 22092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:20.995214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:37211)
0810 19:02:20.995328 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:20.995379 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:02:20.995666 22083 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:02:21.034173 22093 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:21.034410 22093 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:21.034451 22093 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:21.038291 22093 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:21.038340 22093 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:21.038395 22093 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:02:21.043514 22093 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:44475
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=43051
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22093
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:21.044420 22093 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:21.045348 22093 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:21.054626 22101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:21.054711 22102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:21.054701 22106 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:21.054848 22093 server_base.cc:1034] running on GCE node
I20260810 19:02:21.055379 22093 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:21.055975 22093 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:21.057164 22093 hybrid_clock.cc:648] HybridClock initialized: now 1786388541057151 us; error 25 us; skew 500 ppm
Aug 10 19:02:21 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388541, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:21.059835 22093 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:02:21.060655 22093 webserver.cc:460] Webserver started at http://127.27.147.130:43051/ using document root <none> and password file <none>
I20260810 19:02:21.061188 22093 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:21.061281 22093 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:21.065166 22093 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:02:21.067191 22117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:21.068358 22093 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:21.068512 22093 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "c0556cee02b34302b1a53bddb0ff90ce"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "faa86205bf82e41fd9ae6b98619af665"
server_key_iv: "80b3ff7bf30066fe3605c9659c30dbb4"
server_key_version: "encryptionkey@0"
I20260810 19:02:21.069029 22093 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:21.087714 22093 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:21.096120 22093 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:21.096323 22093 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:21.096902 22093 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:21.097710 22093 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:21.097784 22093 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:21.097855 22093 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:21.097904 22093 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:21.106765 22093 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:44475
I20260810 19:02:21.106799 22293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:44475 every 8 connection(s)
I20260810 19:02:21.107751 22093 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:02:21.111246 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:21.109736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:33257)
0810 19:02:21.110372 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:21.110479 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":107,"threads_started":1}
I20260810 19:02:21.111296 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 22093
I20260810 19:02:21.111773 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:35945
--local_ip_for_outbound_sockets=127.27.147.131
--tserver_master_addrs=127.27.147.190:38471
--webserver_port=40217
--webserver_interface=127.27.147.131
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:43805
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:21.111819 22294 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38471 (0 consecutive failures): Network error: Failed to ping master at 127.27.147.190:38471: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
I20260810 19:02:21.112430 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:21.112198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:33913)
0810 19:02:21.112312 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:21.112363 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260810 19:02:21.113137 22301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:21.112942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:35701)
0810 19:02:21.113049 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:21.113090 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:02:21.113373 22294 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 19:02:21.225808 22302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:21.226048 22302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:21.226089 22302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:21.230095 22302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:21.230149 22302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:21.230239 22302 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:21.235421 22302 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:43805
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:35945
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=40217
--tserver_master_addrs=127.27.147.190:38471
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22302
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:21.236703 22302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:21.238068 22302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:21.252660 22310 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:21.252831 22302 server_base.cc:1034] running on GCE node
W20260810 19:02:21.252941 22312 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:21.253098 22309 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:21.253522 22302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:21.254360 22302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:21.255594 22302 hybrid_clock.cc:648] HybridClock initialized: now 1786388541255561 us; error 45 us; skew 500 ppm
Aug 10 19:02:21 dist-test-slave-znh6 krb5kdc[19946](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388541, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:21.259552 22302 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:02:21.260818 22302 webserver.cc:460] Webserver started at http://127.27.147.131:40217/ using document root <none> and password file <none>
I20260810 19:02:21.261615 22302 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:21.261727 22302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:21.267373 22302 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:02:21.270469 22323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:21.272255 22302 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:02:21.272408 22302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00"
format_stamp: "Formatted at 2026-08-10 19:01:38 on dist-test-slave-znh6"
server_key: "fe573f7086a74540be878a3e249ffc9e"
server_key_iv: "daf0d21456279f20a398ae58d2a7fdff"
server_key_version: "encryptionkey@0"
I20260810 19:02:21.273283 22302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:21.296597 22302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:21.299806 22302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:21.300046 22302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:21.300930 22302 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:21.302218 22302 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:21.302285 22302 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:21.302376 22302 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:21.302413 22302 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:21.316677 22302 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:35945
I20260810 19:02:21.316684 22511 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:35945 every 8 connection(s)
I20260810 19:02:21.318092 22302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:02:21.319501 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 22302
I20260810 19:02:21.323675 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:21.320936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:52121)
0810 19:02:21.322483 (+ 1547us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:21.322629 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1283,"thread_start_us":138,"threads_started":1}
W20260810 19:02:21.324158 22512 heartbeater.cc:644] Failed to heartbeat to 127.27.147.190:38471 (0 consecutive failures): Network error: Failed to ping master at 127.27.147.190:38471: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
I20260810 19:02:21.324812 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:21.324525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:41463)
0810 19:02:21.324662 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:21.324729 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 19:02:21.325546 22522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:21.325271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:48407)
0810 19:02:21.325398 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:21.325467 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:02:21.325733 22512 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 19:02:21.996654 22523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:21.996227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:50147)
0810 19:02:21.996484 (+ 257us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:21.996539 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":73,"threads_started":1}
I20260810 19:02:22.114384 22524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:22.113904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:49943)
0810 19:02:22.114240 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:22.114295 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":70,"threads_started":1}
I20260810 19:02:22.326714 22525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:22.326322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:43845)
0810 19:02:22.326576 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:22.326626 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":101,"threads_started":1}
I20260810 19:02:22.766690 22526 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:22.767059 22526 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:22.768239 22526 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
Aug 10 19:02:22 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.130@KRBTEST.COM
Aug 10 19:02:22 dist-test-slave-znh6 krb5kdc[19946](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388540, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.131@KRBTEST.COM
I20260810 19:02:22.781963 22527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:22.768377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:44475 (local address 127.27.147.129:44879)
0810 19:02:22.768716 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:22.768733 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:22.768845 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:22.770373 (+ 1528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:22.770379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:22.770966 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:22.771420 (+ 454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:22.771435 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.772815 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.772817 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:22.773818 (+ 1001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:22.773828 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.774542 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.774552 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:22.774757 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:22.775525 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:22.775537 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:22.777134 (+ 1597us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:22.779549 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:22.779554 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:22.779566 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:22.779876 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:22.780269 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:22.780272 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:22.780274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:22.780419 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:22.781016 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:22.781023 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:22.781253 (+ 230us) client_negotiation.cc:769] Sending connection context
0810 19:02:22.781437 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 19:02:22.781746 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271,"mutex_wait_us":173,"spinlock_wait_cycles":6144,"thread_start_us":62,"threads_started":1}
I20260810 19:02:22.781982 22529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:22.768564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:44879 (local address 127.27.147.130:44475)
0810 19:02:22.768876 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:22.768894 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:22.768939 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:02:22.769665 (+ 726us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:22.769671 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:22.770007 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:22.770308 (+ 301us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:22.771621 (+ 1313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.772656 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.773950 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.774415 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.774591 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:22.777354 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:22.777359 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:22.777370 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:22.777395 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:22.779426 (+ 2031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:22.779998 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:22.780004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:22.780009 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:22.780177 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:22.780517 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:22.780522 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:22.780523 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:22.780890 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:22.781043 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:22.781550 (+ 507us) server_negotiation.cc:299] Negotiation successful
0810 19:02:22.781799 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":59,"threads_started":1}
I20260810 19:02:22.782014 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:22.768490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:35945 (local address 127.27.147.129:56975)
0810 19:02:22.768938 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:22.768952 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:22.769056 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:22.770373 (+ 1317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:22.770379 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:22.770964 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:22.771420 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:22.771434 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.772777 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.772782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:22.773819 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:22.773829 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.774543 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.774547 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:22.774756 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:22.775464 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:22.775480 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:22.777496 (+ 2016us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:22.779580 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:22.779585 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:22.779587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:22.779877 (+ 290us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:22.780285 (+ 408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:22.780289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:22.780291 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:22.780427 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:22.781017 (+ 590us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:22.781023 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:22.781252 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 19:02:22.781437 (+ 185us) client_negotiation.cc:240] Negotiation successful
0810 19:02:22.781833 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":277,"spinlock_wait_cycles":33664,"thread_start_us":74,"threads_started":1}
I20260810 19:02:22.782657 22530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:22.768649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:56975 (local address 127.27.147.131:35945)
0810 19:02:22.769070 (+ 421us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:22.769091 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:22.769125 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:02:22.769744 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:22.769749 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:22.770032 (+ 283us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:22.770317 (+ 285us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:22.771594 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.772634 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.773953 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:22.774414 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:22.774592 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:22.777697 (+ 3105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:22.777702 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:22.777713 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:22.777739 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:22.779467 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:22.779995 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:22.780007 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:22.780012 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:22.780173 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:22.780525 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:22.780528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:22.780530 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:22.780899 (+ 369us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:22.781043 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:22.781563 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 19:02:22.781831 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":94,"threads_started":1}
W20260810 19:02:22.784533 21911 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
W20260810 19:02:22.784693 21913 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
I20260810 19:02:22.784781 21913 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7ea371eba4948fe8503a51fa8dfec4a; no voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce
I20260810 19:02:22.785055 22526 raft_consensus.cc:2726] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:02:22.997669 22528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:22.997407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:54657)
0810 19:02:22.997532 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:22.997585 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260810 19:02:23.115545 22531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:23.115136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:34613)
0810 19:02:23.115402 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:23.115453 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":71,"threads_started":1}
I20260810 19:02:23.327867 22532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:23.327443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:44533)
0810 19:02:23.327719 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:23.327774 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":76,"threads_started":1}
I20260810 19:02:23.998939 22536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:23.998431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:55717)
0810 19:02:23.998711 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:23.998774 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":78,"threads_started":1}
I20260810 19:02:24.116673 22537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:24.116251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:39091)
0810 19:02:24.116524 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:24.116580 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":91,"threads_started":1}
I20260810 19:02:24.328969 22538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:24.328586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:50769)
0810 19:02:24.328818 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:24.328870 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260810 19:02:24.980834 22540 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:24.980996 22540 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:24.981484 22540 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
W20260810 19:02:24.982270 21913 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
W20260810 19:02:24.982360 21911 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
I20260810 19:02:24.982491 21911 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7ea371eba4948fe8503a51fa8dfec4a; no voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce
I20260810 19:02:24.982712 22540 raft_consensus.cc:2726] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:02:25.000023 22541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:24.999656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:60415)
0810 19:02:24.999888 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:24.999939 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":70,"threads_started":1}
I20260810 19:02:25.117761 22542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:25.117389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:50263)
0810 19:02:25.117625 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:25.117674 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":76,"threads_started":1}
I20260810 19:02:25.330044 22543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:25.329655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:58079)
0810 19:02:25.329909 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:25.329958 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":75,"threads_started":1}
I20260810 19:02:26.001343 22545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:26.000900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:39351)
0810 19:02:26.001181 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:26.001237 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":71,"threads_started":1}
I20260810 19:02:26.118965 22546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:26.118500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:33981)
0810 19:02:26.118762 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:26.118868 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":73,"threads_started":1}
I20260810 19:02:26.331224 22547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:26.330795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:49019)
0810 19:02:26.331083 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:26.331136 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":220,"thread_start_us":84,"threads_started":1}
I20260810 19:02:26.579658 22549 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:26.579802 22549 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:26.580350 22549 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
W20260810 19:02:26.581176 21911 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
W20260810 19:02:26.581326 21913 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
I20260810 19:02:26.581418 21913 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7ea371eba4948fe8503a51fa8dfec4a; no voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce
I20260810 19:02:26.581583 22549 raft_consensus.cc:2726] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:02:27.002550 22550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:27.002092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:41815)
0810 19:02:27.002392 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:27.002449 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":76,"threads_started":1}
I20260810 19:02:27.120100 22551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:27.119688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:40729)
0810 19:02:27.119953 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:27.120008 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":76,"threads_started":1}
I20260810 19:02:27.332329 22553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:27.331921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:53881)
0810 19:02:27.332193 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:27.332244 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260810 19:02:28.003818 22554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:28.003379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:48951)
0810 19:02:28.003657 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:28.003719 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":78,"threads_started":1}
I20260810 19:02:28.121372 22557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:28.120918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:56217)
0810 19:02:28.121227 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:28.121282 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":241,"thread_start_us":71,"threads_started":1}
I20260810 19:02:28.333614 22558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:28.333122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:46465)
0810 19:02:28.333446 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:28.333510 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":252,"thread_start_us":93,"threads_started":1}
I20260810 19:02:28.436504 22560 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:28.436645 22560 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:28.437140 22560 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
W20260810 19:02:28.437985 21913 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
W20260810 19:02:28.438149 21911 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
I20260810 19:02:28.438223 21911 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7ea371eba4948fe8503a51fa8dfec4a; no voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce
I20260810 19:02:28.438437 22560 raft_consensus.cc:2726] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:02:29.005074 22561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:29.004635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:36301)
0810 19:02:29.004916 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:29.004976 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":74,"threads_started":1}
I20260810 19:02:29.122536 22562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:29.122157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:58411)
0810 19:02:29.122396 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:29.122449 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":72,"threads_started":1}
I20260810 19:02:29.334771 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:29.334360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:44721)
0810 19:02:29.334621 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:29.334675 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":71,"threads_started":1}
I20260810 19:02:30.006498 22565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:30.005977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:54633)
0810 19:02:30.006239 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:30.006359 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":96,"threads_started":1}
I20260810 19:02:30.123770 22568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:30.123376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:35563)
0810 19:02:30.123621 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:30.123676 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177,"thread_start_us":74,"threads_started":1}
I20260810 19:02:30.335918 22570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:30.335534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:43123)
0810 19:02:30.335773 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:30.335826 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":78,"threads_started":1}
I20260810 19:02:31.007839 22572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:31.007440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:52711)
0810 19:02:31.007690 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:31.007742 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":77,"threads_started":1}
I20260810 19:02:31.125001 22573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:31.124557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:54009)
0810 19:02:31.124828 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:31.124904 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":80,"threads_started":1}
I20260810 19:02:31.337262 22574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:31.336746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:49663)
0810 19:02:31.337058 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:31.337144 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":81,"threads_started":1}
I20260810 19:02:32.009045 22575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:32.008617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:48599)
0810 19:02:32.008891 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:32.008945 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":99,"threads_started":1}
I20260810 19:02:32.126227 22576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:32.125800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:38849)
0810 19:02:32.126058 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:32.126113 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":93,"threads_started":1}
I20260810 19:02:32.338409 22579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:32.337994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:57823)
0810 19:02:32.338242 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:32.338295 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":78,"threads_started":1}
I20260810 19:02:32.572379 22580 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:32.572528 22580 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:32.573009 22580 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
W20260810 19:02:32.573879 21913 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
W20260810 19:02:32.574013 21911 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
I20260810 19:02:32.574126 21911 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7ea371eba4948fe8503a51fa8dfec4a; no voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce
I20260810 19:02:32.574338 22580 raft_consensus.cc:2726] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:02:33.010510 22581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:33.010017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:57513)
0810 19:02:33.010334 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:33.010406 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":84,"threads_started":1}
I20260810 19:02:33.127521 22583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:33.127051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:34241)
0810 19:02:33.127348 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:33.127425 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":72,"threads_started":1}
I20260810 19:02:33.339888 22585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:33.339378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:45777)
0810 19:02:33.339716 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:33.339785 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":110,"threads_started":1}
I20260810 19:02:34.011854 22586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:34.011402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:58589)
0810 19:02:34.011696 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:34.011754 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":81,"threads_started":1}
I20260810 19:02:34.128803 22588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:34.128359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:49609)
0810 19:02:34.128647 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:34.128704 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":106,"threads_started":1}
I20260810 19:02:34.341141 22590 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:34.340731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:51131)
0810 19:02:34.340992 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:34.341046 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":76,"threads_started":1}
I20260810 19:02:35.013155 22591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:35.012722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:43995)
0810 19:02:35.013006 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:35.013057 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":218,"thread_start_us":78,"threads_started":1}
I20260810 19:02:35.129910 22592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:35.129512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:57037)
0810 19:02:35.129760 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:35.129817 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":73,"threads_started":1}
I20260810 19:02:35.342394 22593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:35.341947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:52393)
0810 19:02:35.342225 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:35.342284 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":199,"thread_start_us":81,"threads_started":1}
I20260810 19:02:36.014533 22597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:36.014100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.129:54167)
0810 19:02:36.014361 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:36.014412 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":193,"thread_start_us":94,"threads_started":1}
I20260810 19:02:36.098594 22598 raft_consensus.cc:482] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:36.098763 22598 raft_consensus.cc:504] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0556cee02b34302b1a53bddb0ff90ce" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 44475 } } peers { permanent_uuid: "a88d6cfe7bbd4cf9858a973c3d7a9f00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 35945 } } peers { permanent_uuid: "c7ea371eba4948fe8503a51fa8dfec4a" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 44319 } }
I20260810 19:02:36.099215 22598 leader_election.cc:288] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475), a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945)
W20260810 19:02:36.100057 21913 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer a88d6cfe7bbd4cf9858a973c3d7a9f00 (127.27.147.131:35945): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
W20260810 19:02:36.100159 21911 leader_election.cc:341] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0556cee02b34302b1a53bddb0ff90ce (127.27.147.130:44475): Not found: Tablet not found: ee233c92d47f49b1a992e224a021a373
I20260810 19:02:36.100225 21911 leader_election.cc:302] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c7ea371eba4948fe8503a51fa8dfec4a; no voters: a88d6cfe7bbd4cf9858a973c3d7a9f00, c0556cee02b34302b1a53bddb0ff90ce
I20260810 19:02:36.100364 22598 raft_consensus.cc:2726] T ee233c92d47f49b1a992e224a021a373 P c7ea371eba4948fe8503a51fa8dfec4a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 19:02:36.131174 22599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:36.130761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.130:48113)
0810 19:02:36.131030 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:36.131081 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191,"thread_start_us":120,"threads_started":1}
I20260810 19:02:36.323562 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 21815
I20260810 19:02:36.343885 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 22093
I20260810 19:02:36.344034 22601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:36.343459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38471 (local address 127.27.147.131:43027)
0810 19:02:36.343825 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:36.343899 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.190:38471: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":78,"threads_started":1}
I20260810 19:02:36.356907 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 22302
2026-08-10T19:02:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (59689 ms)
[----------] 30 tests from SecurityITest (251075 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:36 dist-test-slave-znh6 krb5kdc[22604](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:36 dist-test-slave-znh6 krb5kdc[22604](info): set up 2 sockets
Aug 10 19:02:36 dist-test-slave-znh6 krb5kdc[22604](info): commencing operation
krb5kdc: starting...
W20260810 19:02:37.611989 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.007s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:37 dist-test-slave-znh6 krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:02:37.742722 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:42713
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:42713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:37.854780 22626 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:37.855095 22626 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:37.855136 22626 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:37.859184 22626 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:37.859233 22626 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:37.859261 22626 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:37.859298 22626 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:37.859334 22626 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:37.864912 22626 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:42713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:42713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22626
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:37.865921 22626 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:37.867040 22626 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:37.875651 22636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:37.875651 22639 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:37.875674 22633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:37.876101 22626 server_base.cc:1034] running on GCE node
I20260810 19:02:37.876603 22626 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:37.877588 22626 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:37.878785 22626 hybrid_clock.cc:648] HybridClock initialized: now 1786388557878771 us; error 26 us; skew 500 ppm
Aug 10 19:02:37 dist-test-slave-znh6 krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388557, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:37.881520 22626 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:02:37.882411 22626 webserver.cc:460] Webserver started at http://127.27.147.190:46261/ using document root <none> and password file <none>
I20260810 19:02:37.883069 22626 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:37.883163 22626 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:37.883411 22626 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:37.885442 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "7b53a426ece1427fa78ebec29d5d78fd"
format_stamp: "Formatted at 2026-08-10 19:02:37 on dist-test-slave-znh6"
server_key: "007af07bf736d2c01cbee3a304d2d4bd"
server_key_iv: "c3e7f2c2e9dfd47f25191882cc4a2caa"
server_key_version: "encryptionkey@0"
I20260810 19:02:37.886039 22626 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "7b53a426ece1427fa78ebec29d5d78fd"
format_stamp: "Formatted at 2026-08-10 19:02:37 on dist-test-slave-znh6"
server_key: "007af07bf736d2c01cbee3a304d2d4bd"
server_key_iv: "c3e7f2c2e9dfd47f25191882cc4a2caa"
server_key_version: "encryptionkey@0"
I20260810 19:02:37.889672 22626 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 19:02:37.892174 22651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:37.893361 22626 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:02:37.893504 22626 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "7b53a426ece1427fa78ebec29d5d78fd"
format_stamp: "Formatted at 2026-08-10 19:02:37 on dist-test-slave-znh6"
server_key: "007af07bf736d2c01cbee3a304d2d4bd"
server_key_iv: "c3e7f2c2e9dfd47f25191882cc4a2caa"
server_key_version: "encryptionkey@0"
I20260810 19:02:37.893617 22626 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:37.904830 22626 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:37.907507 22626 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:37.907711 22626 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:37.914742 22626 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:42713
I20260810 19:02:37.914873 22732 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:42713 every 8 connection(s)
I20260810 19:02:37.915710 22626 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:37.918982 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 22626
I20260810 19:02:37.919113 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:37.919209 22733 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:37.919441 28238 external_mini_cluster.cc:1428] Setting key 2a50da51dd1cf8ea3694c9892ef8fe97
I20260810 19:02:37.926373 22733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd: Bootstrap starting.
Aug 10 19:02:37 dist-test-slave-znh6 krb5kdc[22604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:37.929179 22733 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:37.930115 22733 log.cc:824] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:37.932334 22733 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd: No bootstrap required, opened a new log
I20260810 19:02:37.933959 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:37.920806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33258 (local address 127.27.147.190:42713)
0810 19:02:37.921230 (+ 424us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:37.921275 (+ 45us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:37.921311 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:02:37.921942 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:37.921958 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:37.922272 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:37.922536 (+ 264us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:37.923530 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:37.924685 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:37.925878 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:37.926373 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:37.926557 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:37.929270 (+ 2713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:37.929277 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:37.929293 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:37.929327 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:37.931191 (+ 1864us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:37.931673 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:37.931678 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:37.931682 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:37.931789 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:37.932090 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:37.932092 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:37.932093 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:37.932517 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:37.932661 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:37.932984 (+ 323us) server_negotiation.cc:299] Negotiation successful
0810 19:02:37.933272 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":288,"thread_start_us":153,"threads_started":1}
I20260810 19:02:37.936079 22733 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53a426ece1427fa78ebec29d5d78fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42713 } }
I20260810 19:02:37.936331 22733 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:37.936414 22733 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b53a426ece1427fa78ebec29d5d78fd, State: Initialized, Role: FOLLOWER
I20260810 19:02:37.936825 22733 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53a426ece1427fa78ebec29d5d78fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42713 } }
I20260810 19:02:37.936978 22733 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:37.937065 22733 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:37.937186 22733 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:37.938045 22733 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53a426ece1427fa78ebec29d5d78fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42713 } }
I20260810 19:02:37.938388 22733 leader_election.cc:302] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7b53a426ece1427fa78ebec29d5d78fd; no voters:
I20260810 19:02:37.938714 22733 leader_election.cc:288] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:37.938823 22738 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:37.939107 22738 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [term 1 LEADER]: Becoming Leader. State: Replica: 7b53a426ece1427fa78ebec29d5d78fd, State: Running, Role: LEADER
I20260810 19:02:37.939400 22738 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53a426ece1427fa78ebec29d5d78fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42713 } }
I20260810 19:02:37.939939 22733 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:37.940514 22738 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7b53a426ece1427fa78ebec29d5d78fd. Latest consensus state: current_term: 1 leader_uuid: "7b53a426ece1427fa78ebec29d5d78fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53a426ece1427fa78ebec29d5d78fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42713 } } }
I20260810 19:02:37.940727 22738 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:37.940965 22741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7b53a426ece1427fa78ebec29d5d78fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7b53a426ece1427fa78ebec29d5d78fd" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 42713 } } }
I20260810 19:02:37.941120 22741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:37.941229 22752 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:37.944895 22752 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:37.950682 22752 catalog_manager.cc:1347] Generated new cluster ID: 1454b1fe328b47a297a5c8bd4eff9658
I20260810 19:02:37.950757 22752 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:37.962028 22752 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:37.963153 22752 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:37.970500 22752 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd: Generated new TSK 0
I20260810 19:02:37.971251 22752 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 19:02:38.006880 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42713
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:38.121487 22773 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:38.121766 22773 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:38.121845 22773 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:38.125968 22773 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:38.126049 22773 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:38.126140 22773 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:02:38.131419 22773 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22773
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:38.132452 22773 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:38.133469 22773 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:38.143566 22782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:38.143658 22773 server_base.cc:1034] running on GCE node
W20260810 19:02:38.143584 22781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:38.143587 22789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:38.144212 22773 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:38.144774 22773 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:38.145977 22773 hybrid_clock.cc:648] HybridClock initialized: now 1786388558145946 us; error 42 us; skew 500 ppm
Aug 10 19:02:38 dist-test-slave-znh6 krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388558, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:38.148734 22773 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:02:38.149559 22773 webserver.cc:460] Webserver started at http://127.27.147.129:45855/ using document root <none> and password file <none>
I20260810 19:02:38.150113 22773 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:38.150204 22773 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:38.150461 22773 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:38.152526 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "92166c3154a342618227e85623ad57c8"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "4f1ed774678c3c76028607fc06e0f57c"
server_key_iv: "5cedcad9e2e455a68b8a98541fba7305"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.153090 22773 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "92166c3154a342618227e85623ad57c8"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "4f1ed774678c3c76028607fc06e0f57c"
server_key_iv: "5cedcad9e2e455a68b8a98541fba7305"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.156752 22773 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 19:02:38.159196 22802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.160444 22773 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:02:38.160607 22773 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "92166c3154a342618227e85623ad57c8"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "4f1ed774678c3c76028607fc06e0f57c"
server_key_iv: "5cedcad9e2e455a68b8a98541fba7305"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.160722 22773 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:38.175835 22773 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:38.178390 22773 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:38.178591 22773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:38.179180 22773 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:38.179976 22773 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:38.180061 22773 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.180138 22773 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:38.180194 22773 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.189253 22773 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:38441
I20260810 19:02:38.189281 22992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:38441 every 8 connection(s)
I20260810 19:02:38.190086 22773 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:02:38.193113 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 22773
I20260810 19:02:38.193265 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:02:38.193564 28238 external_mini_cluster.cc:1428] Setting key 6534fd5e4da6165c28ac2dd62ccadf56
Aug 10 19:02:38 dist-test-slave-znh6 krb5kdc[22604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388558, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 19:02:38.204909 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.191842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:34629 (local address 127.27.147.190:42713)
0810 19:02:38.191979 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.191982 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.192759 (+ 777us) server_negotiation.cc:407] Connection header received
0810 19:02:38.194271 (+ 1512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.194275 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.194335 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.194442 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:38.195692 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.196234 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.197818 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.198251 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.198357 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.201170 (+ 2813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:38.201174 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:38.201177 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:38.201211 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:38.202770 (+ 1559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.203342 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.203346 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.203348 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.203441 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.203795 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.203800 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.203802 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.203996 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:38.204110 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.204601 (+ 491us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.204735 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:02:38.205690 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.192161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42713 (local address 127.27.147.129:34629)
0810 19:02:38.192590 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.192630 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.194052 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.194580 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.194587 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.195040 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:38.195539 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.195553 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.196400 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.196403 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.197667 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.197683 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.198424 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.198430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.198655 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.199271 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:38.199296 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:38.201001 (+ 1705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:38.202933 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:38.202939 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:38.202953 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:38.203216 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:38.203541 (+ 325us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:38.203545 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:38.203547 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:38.203692 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:38.204109 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:38.204116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:38.204351 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.204544 (+ 193us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.204854 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"thread_start_us":116,"threads_started":1}
I20260810 19:02:38.207055 22993 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42713
I20260810 19:02:38.207293 22993 heartbeater.cc:464] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 19:02:38.207901 22993 heartbeater.cc:511] Master 127.27.147.190:42713 requested a full tablet report, sending...
I20260810 19:02:38.209450 22678 ts_manager.cc:194] Registered new tserver with Master: 92166c3154a342618227e85623ad57c8 (127.27.147.129:38441)
I20260810 19:02:38.210731 22678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:34629
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 19:02:38.224062 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42713
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:38.335588 22998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:38.335808 22998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:38.335847 22998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:38.339887 22998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:38.339942 22998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:38.339998 22998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:02:38.345247 22998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:38.346117 22998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:38.347076 22998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:38.356405 23015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:38.356364 23011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:38.356521 22998 server_base.cc:1034] running on GCE node
W20260810 19:02:38.356370 23013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:38.357067 22998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:38.357681 22998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:38.358896 22998 hybrid_clock.cc:648] HybridClock initialized: now 1786388558358871 us; error 28 us; skew 500 ppm
Aug 10 19:02:38 dist-test-slave-znh6 krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388558, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:38.361599 22998 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:02:38.362416 22998 webserver.cc:460] Webserver started at http://127.27.147.130:42161/ using document root <none> and password file <none>
I20260810 19:02:38.363035 22998 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:38.363126 22998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:38.363380 22998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:38.365469 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "a4ce95e7467541fc93f4ee29543e6bcf"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "0bfb374b6d441e21984c54570ff622d0"
server_key_iv: "5dffe9c753dc21de2d09d29549900f79"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.366034 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "a4ce95e7467541fc93f4ee29543e6bcf"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "0bfb374b6d441e21984c54570ff622d0"
server_key_iv: "5dffe9c753dc21de2d09d29549900f79"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.369721 22998 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 19:02:38.372162 23029 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.373343 22998 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:02:38.373487 22998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "a4ce95e7467541fc93f4ee29543e6bcf"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "0bfb374b6d441e21984c54570ff622d0"
server_key_iv: "5dffe9c753dc21de2d09d29549900f79"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.373596 22998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:38.397445 22998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:38.400689 22998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:38.400918 22998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:38.401755 22998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:38.402973 22998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:38.403056 22998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.403131 22998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:38.403169 22998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.414160 22998 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:40183
I20260810 19:02:38.414224 23218 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:40183 every 8 connection(s)
I20260810 19:02:38.415143 22998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:02:38.420143 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 22998
I20260810 19:02:38.420317 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:02:38.420614 28238 external_mini_cluster.cc:1428] Setting key 21d11d61476e340bb2667e7d25dc08fa
Aug 10 19:02:38 dist-test-slave-znh6 krb5kdc[22604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388558, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:38.429216 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.416910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:51433 (local address 127.27.147.190:42713)
0810 19:02:38.417047 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.417052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.417864 (+ 812us) server_negotiation.cc:407] Connection header received
0810 19:02:38.419302 (+ 1438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.419306 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.419366 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.419485 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:38.420686 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.421237 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.422363 (+ 1126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.422597 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.422710 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.425171 (+ 2461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:38.425175 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:38.425177 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:38.425201 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:38.426977 (+ 1776us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.427510 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.427517 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.427522 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.427626 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.427987 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.427990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.427992 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.428237 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:38.428350 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.428989 (+ 639us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.429098 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:02:38.430476 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.417279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42713 (local address 127.27.147.130:51433)
0810 19:02:38.417719 (+ 440us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.417761 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.419093 (+ 1332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.419651 (+ 558us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.419658 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.419990 (+ 332us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:38.420538 (+ 548us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.420553 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.421371 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.421379 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.422232 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.422241 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.422760 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.422764 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.422981 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.423564 (+ 583us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:38.423581 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:38.424995 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:38.427110 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:38.427115 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:38.427126 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:38.427381 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:38.427736 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:38.427739 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:38.427740 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:38.427880 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:38.428362 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:38.428371 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:38.428688 (+ 317us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.428932 (+ 244us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.429295 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":119,"threads_started":1}
I20260810 19:02:38.431727 23219 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42713
I20260810 19:02:38.431968 23219 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:38.432430 23219 heartbeater.cc:511] Master 127.27.147.190:42713 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 19:02:38.433422 22678 ts_manager.cc:194] Registered new tserver with Master: a4ce95e7467541fc93f4ee29543e6bcf (127.27.147.130:40183)
I20260810 19:02:38.434077 22678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:51433
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 19:02:38.449718 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42713
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42317
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:38.561033 23229 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:38.561421 23229 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:38.561510 23229 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:38.565850 23229 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:38.565927 23229 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:38.566018 23229 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:38.571200 23229 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42317
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:42713
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23229
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:38.572230 23229 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:38.573221 23229 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:38.583223 23240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:38.583236 23241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:38.583212 23243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:38.583449 23229 server_base.cc:1034] running on GCE node
I20260810 19:02:38.583858 23229 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:38.584436 23229 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:38.585598 23229 hybrid_clock.cc:648] HybridClock initialized: now 1786388558585581 us; error 34 us; skew 500 ppm
Aug 10 19:02:38 dist-test-slave-znh6 krb5kdc[22604](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388558, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:38.588370 23229 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:02:38.589202 23229 webserver.cc:460] Webserver started at http://127.27.147.131:34911/ using document root <none> and password file <none>
I20260810 19:02:38.589772 23229 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:38.589870 23229 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:38.590111 23229 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:38.592160 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "f11737f061274b458b50ac70b893502c"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "aa050ba5736cee90ce25664e7bb6f82a"
server_key_iv: "db2acd609cb2556adc16d739882c8e96"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.592732 23229 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "f11737f061274b458b50ac70b893502c"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "aa050ba5736cee90ce25664e7bb6f82a"
server_key_iv: "db2acd609cb2556adc16d739882c8e96"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.596396 23229 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:02:38.598762 23255 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.599973 23229 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:02:38.600112 23229 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "f11737f061274b458b50ac70b893502c"
format_stamp: "Formatted at 2026-08-10 19:02:38 on dist-test-slave-znh6"
server_key: "aa050ba5736cee90ce25664e7bb6f82a"
server_key_iv: "db2acd609cb2556adc16d739882c8e96"
server_key_version: "encryptionkey@0"
I20260810 19:02:38.600222 23229 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:38.625872 23229 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:38.628592 23229 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:38.628793 23229 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:38.629357 23229 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:38.630156 23229 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:38.630230 23229 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.630303 23229 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:38.630352 23229 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:38.639436 23229 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:42377
I20260810 19:02:38.639464 23447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:42377 every 8 connection(s)
I20260810 19:02:38.640373 23229 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:02:38.645509 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 23229
I20260810 19:02:38.645655 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:02:38.646027 28238 external_mini_cluster.cc:1428] Setting key 802f218f5946c4bae40f4c64519cd200
Aug 10 19:02:38 dist-test-slave-znh6 krb5kdc[22604](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388558, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:38.653954 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.642146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:41211 (local address 127.27.147.190:42713)
0810 19:02:38.642294 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.642299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.643211 (+ 912us) server_negotiation.cc:407] Connection header received
0810 19:02:38.644507 (+ 1296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.644510 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.644558 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.644632 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:38.645987 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.646567 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.648078 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.648351 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.648442 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.650856 (+ 2414us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:38.650869 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:38.650872 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:38.650895 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:38.652126 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.652583 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.652586 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.652588 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.652665 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.653033 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.653035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.653036 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.653200 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:38.653277 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.653737 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.653832 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:02:38.654754 23452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.642585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:42713 (local address 127.27.147.131:41211)
0810 19:02:38.643065 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.643108 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.644310 (+ 1202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.644791 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.644798 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.645120 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:38.645828 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.645851 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.646694 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.646698 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.647882 (+ 1184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.647892 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.648485 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.648488 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.648626 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.649234 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:38.649250 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:38.650665 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:38.652230 (+ 1565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:38.652234 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:38.652245 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:38.652492 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:38.652791 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:38.652794 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:38.652795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:38.652940 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:38.653300 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:38.653305 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:38.653532 (+ 227us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.653688 (+ 156us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.653946 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":111,"threads_started":1}
I20260810 19:02:38.655937 23449 heartbeater.cc:347] Connected to a master server at 127.27.147.190:42713
I20260810 19:02:38.656185 23449 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:38.656666 23449 heartbeater.cc:511] Master 127.27.147.190:42713 requested a full tablet report, sending...
I20260810 19:02:38.657689 22678 ts_manager.cc:194] Registered new tserver with Master: f11737f061274b458b50ac70b893502c (127.27.147.131:42377)
I20260810 19:02:38.658269 22678 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:41211
I20260810 19:02:38.660131 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:02:38.669570 22736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.662106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33266 (local address 127.27.147.190:42713)
0810 19:02:38.662235 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.662242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.662471 (+ 229us) server_negotiation.cc:407] Connection header received
0810 19:02:38.662620 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.662624 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.662670 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.662765 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:38.663496 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.663992 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.664871 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.665086 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.665194 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.666501 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:38.666505 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:38.666507 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:38.666529 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:38.667874 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.668320 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.668324 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.668325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.668408 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:38.668692 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:38.668696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:38.668698 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:38.668905 (+ 207us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:38.669056 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.669304 (+ 248us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.669416 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 19:02:38.672164 22678 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33266:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:02:38.675143 22678 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:02:38.689265 23470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.683944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:40183 (local address 127.0.0.1:48218)
0810 19:02:38.684669 (+ 725us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.684684 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.684768 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.685478 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.685481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.685501 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:38.685587 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.685593 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.686871 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.686877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.687886 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.687899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.688708 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.688712 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.688829 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.688869 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.688997 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.689136 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"mutex_wait_us":3,"thread_start_us":110,"threads_started":1}
I20260810 19:02:38.689280 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.684279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48218 (local address 127.27.147.130:40183)
0810 19:02:38.685012 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.685033 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.685057 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:02:38.685129 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.685134 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.685320 (+ 186us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.685476 (+ 156us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:38.685709 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.686723 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.688014 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.688577 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.688719 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.688852 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.689028 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.689123 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":101,"threads_started":1}
I20260810 19:02:38.689519 23467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.683190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:38441 (local address 127.0.0.1:45974)
0810 19:02:38.683782 (+ 592us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.683835 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.683952 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.684794 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.684802 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.684825 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:38.684957 (+ 132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.684966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.686562 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.686565 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.687710 (+ 1145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.687722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.689020 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.689024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.689178 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.689222 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.689306 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.689389 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":424,"mutex_wait_us":30,"thread_start_us":124,"threads_started":1}
I20260810 19:02:38.689604 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.683838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.27.147.131:42377)
0810 19:02:38.684238 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.684254 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.684287 (+ 33us) server_negotiation.cc:407] Connection header received
0810 19:02:38.684461 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.684466 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.684659 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.684818 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:38.685271 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.686155 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.687865 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.688706 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.688833 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.688951 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.689379 (+ 428us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.689469 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":151,"threads_started":1}
I20260810 19:02:38.689857 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.683335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.27.147.129:38441)
0810 19:02:38.684244 (+ 909us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.684261 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.684285 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:02:38.684373 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.684383 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.684597 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.685122 (+ 525us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:38.685261 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.686421 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.687865 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.688706 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.688833 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.688951 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.689616 (+ 665us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.689725 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":67,"threads_started":1}
I20260810 19:02:38.689930 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.683367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:42377 (local address 127.0.0.1:44266)
0810 19:02:38.684208 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.684223 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.684317 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.684962 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.684965 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.684987 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:38.685107 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.685114 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.686293 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.686299 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.687710 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.687722 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.689020 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.689024 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.689183 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.689229 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.689299 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.689798 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":776,"mutex_wait_us":469,"thread_start_us":93,"threads_started":1}
I20260810 19:02:38.692560 23114 tablet_service.cc:1467] Processing CreateTablet for tablet 75593be963b34cf08b6d7782905e4b4b (DEFAULT_TABLE table=test-table [id=f07c55d22997416893b07ae8b380ed2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:38.692700 22885 tablet_service.cc:1467] Processing CreateTablet for tablet 75593be963b34cf08b6d7782905e4b4b (DEFAULT_TABLE table=test-table [id=f07c55d22997416893b07ae8b380ed2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:38.693184 23332 tablet_service.cc:1467] Processing CreateTablet for tablet 75593be963b34cf08b6d7782905e4b4b (DEFAULT_TABLE table=test-table [id=f07c55d22997416893b07ae8b380ed2c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:38.693586 23114 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75593be963b34cf08b6d7782905e4b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:38.693755 22885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75593be963b34cf08b6d7782905e4b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:38.694121 23332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 75593be963b34cf08b6d7782905e4b4b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:38.699961 23472 tablet_bootstrap.cc:492] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Bootstrap starting.
I20260810 19:02:38.701363 23473 tablet_bootstrap.cc:492] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: Bootstrap starting.
I20260810 19:02:38.702355 23472 tablet_bootstrap.cc:654] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:38.702360 23474 tablet_bootstrap.cc:492] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Bootstrap starting.
I20260810 19:02:38.703459 23473 tablet_bootstrap.cc:654] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:38.703881 23472 log.cc:824] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:38.704579 23474 tablet_bootstrap.cc:654] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:38.704721 23473 log.cc:824] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:38.705817 23474 log.cc:824] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:38.706043 23472 tablet_bootstrap.cc:492] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: No bootstrap required, opened a new log
I20260810 19:02:38.706282 23472 ts_tablet_manager.cc:1397] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Time spent bootstrapping tablet: real 0.006s user 0.006s sys 0.000s
I20260810 19:02:38.706876 23473 tablet_bootstrap.cc:492] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: No bootstrap required, opened a new log
I20260810 19:02:38.707129 23473 ts_tablet_manager.cc:1397] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:02:38.707863 23474 tablet_bootstrap.cc:492] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: No bootstrap required, opened a new log
I20260810 19:02:38.708041 23474 ts_tablet_manager.cc:1397] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 19:02:38.709362 23472 raft_consensus.cc:348] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.709594 23472 raft_consensus.cc:374] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:38.709671 23472 raft_consensus.cc:729] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4ce95e7467541fc93f4ee29543e6bcf, State: Initialized, Role: FOLLOWER
I20260810 19:02:38.710129 23472 consensus_queue.cc:260] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.710770 23219 heartbeater.cc:503] Master 127.27.147.190:42713 was elected leader, sending a full tablet report...
I20260810 19:02:38.710911 23473 raft_consensus.cc:348] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.711144 23473 raft_consensus.cc:374] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:38.711151 23472 ts_tablet_manager.cc:1428] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:02:38.711193 23473 raft_consensus.cc:729] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92166c3154a342618227e85623ad57c8, State: Initialized, Role: FOLLOWER
I20260810 19:02:38.711625 23473 consensus_queue.cc:260] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.711728 23474 raft_consensus.cc:348] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.712169 23474 raft_consensus.cc:374] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:38.712278 23474 raft_consensus.cc:729] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f11737f061274b458b50ac70b893502c, State: Initialized, Role: FOLLOWER
I20260810 19:02:38.712608 23473 ts_tablet_manager.cc:1428] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:02:38.712745 23474 consensus_queue.cc:260] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.713033 22993 heartbeater.cc:503] Master 127.27.147.190:42713 was elected leader, sending a full tablet report...
I20260810 19:02:38.713434 23449 heartbeater.cc:503] Master 127.27.147.190:42713 was elected leader, sending a full tablet report...
I20260810 19:02:38.713721 23474 ts_tablet_manager.cc:1428] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Time spent starting tablet: real 0.006s user 0.002s sys 0.003s
W20260810 19:02:38.891489 23451 tablet.cc:2369] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:02:38.916260 23220 tablet.cc:2369] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 19:02:38.941850 22994 tablet.cc:2369] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:02:38.974633 23481 raft_consensus.cc:482] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:38.974877 23481 raft_consensus.cc:504] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.975988 23481 leader_election.cc:288] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f11737f061274b458b50ac70b893502c (127.27.147.131:42377), a4ce95e7467541fc93f4ee29543e6bcf (127.27.147.130:40183)
I20260810 19:02:38.980159 23483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.976244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:42377 (local address 127.27.147.129:39285)
0810 19:02:38.976567 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.976584 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.976703 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.977213 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.977215 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.977244 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:38.977347 (+ 103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.977351 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.978089 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.978092 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.979026 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.979036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.979703 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.979707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.979809 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.979829 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.979926 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.979997 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":145,"threads_started":1}
I20260810 19:02:38.980146 23469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.976341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:39285 (local address 127.27.147.131:42377)
0810 19:02:38.976493 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.976497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.976672 (+ 175us) server_negotiation.cc:407] Connection header received
0810 19:02:38.976867 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.976870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.976955 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.977002 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:38.977464 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.977971 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.979155 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.979568 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.979703 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.979757 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.979942 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.980012 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:02:38.980551 23484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.976686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:40183 (local address 127.27.147.129:42127)
0810 19:02:38.977033 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:38.977046 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:38.977175 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:38.977431 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:38.977434 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:38.977451 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:38.977582 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.977588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.978468 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.978473 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.979338 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:38.979349 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.980142 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.980145 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:38.980248 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.980274 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:02:38.980347 (+ 73us) client_negotiation.cc:240] Negotiation successful
0810 19:02:38.980408 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":110,"threads_started":1}
I20260810 19:02:38.980602 23471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:38.976833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:42127 (local address 127.27.147.130:40183)
0810 19:02:38.976967 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:38.976972 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:38.977166 (+ 194us) server_negotiation.cc:407] Connection header received
0810 19:02:38.977272 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:38.977275 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:38.977323 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:38.977420 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:38.977702 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.978352 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.979471 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:38.980018 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:38.980146 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:38.980194 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:38.980384 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 19:02:38.980473 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:02:38.980978 23366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75593be963b34cf08b6d7782905e4b4b" candidate_uuid: "92166c3154a342618227e85623ad57c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11737f061274b458b50ac70b893502c" is_pre_election: true
I20260810 19:02:38.981283 23366 raft_consensus.cc:2454] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92166c3154a342618227e85623ad57c8 in term 0.
I20260810 19:02:38.981820 22807 leader_election.cc:302] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92166c3154a342618227e85623ad57c8, f11737f061274b458b50ac70b893502c; no voters:
I20260810 19:02:38.982129 23481 raft_consensus.cc:2781] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:02:38.982110 23145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75593be963b34cf08b6d7782905e4b4b" candidate_uuid: "92166c3154a342618227e85623ad57c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" is_pre_election: true
I20260810 19:02:38.982231 23481 raft_consensus.cc:482] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:02:38.982280 23481 raft_consensus.cc:3037] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:38.982398 23145 raft_consensus.cc:2454] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 92166c3154a342618227e85623ad57c8 in term 0.
I20260810 19:02:38.983538 23481 raft_consensus.cc:504] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.983829 23481 leader_election.cc:288] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [CANDIDATE]: Term 1 election: Requested vote from peers f11737f061274b458b50ac70b893502c (127.27.147.131:42377), a4ce95e7467541fc93f4ee29543e6bcf (127.27.147.130:40183)
I20260810 19:02:38.984222 23366 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75593be963b34cf08b6d7782905e4b4b" candidate_uuid: "92166c3154a342618227e85623ad57c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f11737f061274b458b50ac70b893502c"
I20260810 19:02:38.984324 23145 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "75593be963b34cf08b6d7782905e4b4b" candidate_uuid: "92166c3154a342618227e85623ad57c8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4ce95e7467541fc93f4ee29543e6bcf"
I20260810 19:02:38.984375 23366 raft_consensus.cc:3037] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:38.984427 23145 raft_consensus.cc:3037] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:38.985391 23145 raft_consensus.cc:2454] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92166c3154a342618227e85623ad57c8 in term 1.
I20260810 19:02:38.985591 23366 raft_consensus.cc:2454] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 92166c3154a342618227e85623ad57c8 in term 1.
I20260810 19:02:38.985718 22807 leader_election.cc:302] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 92166c3154a342618227e85623ad57c8, a4ce95e7467541fc93f4ee29543e6bcf; no voters:
I20260810 19:02:38.985903 23481 raft_consensus.cc:2781] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:38.986150 23481 raft_consensus.cc:686] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [term 1 LEADER]: Becoming Leader. State: Replica: 92166c3154a342618227e85623ad57c8, State: Running, Role: LEADER
I20260810 19:02:38.986477 23481 consensus_queue.cc:237] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } }
I20260810 19:02:38.989748 22675 catalog_manager.cc:5495] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 92166c3154a342618227e85623ad57c8 (127.27.147.129). New cstate: current_term: 1 leader_uuid: "92166c3154a342618227e85623ad57c8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "92166c3154a342618227e85623ad57c8" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 38441 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 } health_report { overall_health: UNKNOWN } } }
I20260810 19:02:39.050282 23466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.046575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45976 (local address 127.27.147.129:38441)
0810 19:02:39.046732 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:39.046737 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:39.046754 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:02:39.046880 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:39.046884 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:39.046947 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:39.047069 (+ 122us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:39.047346 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.047872 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.048769 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.048997 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.049115 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.049458 (+ 343us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:39.049866 (+ 408us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:02:39.050011 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:39.050078 (+ 67us) server_negotiation.cc:299] Negotiation successful
0810 19:02:39.050134 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:02:39.056190 23366 raft_consensus.cc:1261] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 1 FOLLOWER]: Refusing update from remote peer 92166c3154a342618227e85623ad57c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:39.056190 23145 raft_consensus.cc:1261] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 1 FOLLOWER]: Refusing update from remote peer 92166c3154a342618227e85623ad57c8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:39.056983 23481 consensus_queue.cc:1035] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4ce95e7467541fc93f4ee29543e6bcf" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:39.057149 23486 consensus_queue.cc:1035] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f11737f061274b458b50ac70b893502c" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 42377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:39.066234 23496 mvcc.cc:205] Tried to move back new op lower bound from 7317047537886519296 to 7317047537611542528. Current Snapshot: MvccSnapshot[applied={T|T < 7317047537886519296}]
I20260810 19:02:39.072017 22676 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:39.072193 22676 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33266:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:39.074658 22676 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7b53a426ece1427fa78ebec29d5d78fd: Sending DeleteTablet for 3 replicas of tablet 75593be963b34cf08b6d7782905e4b4b
I20260810 19:02:39.075466 22885 tablet_service.cc:1514] Processing DeleteTablet for tablet 75593be963b34cf08b6d7782905e4b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:39 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:45974
I20260810 19:02:39.075559 23332 tablet_service.cc:1514] Processing DeleteTablet for tablet 75593be963b34cf08b6d7782905e4b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:39 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:44266
I20260810 19:02:39.075636 23114 tablet_service.cc:1514] Processing DeleteTablet for tablet 75593be963b34cf08b6d7782905e4b4b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:39 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:48218
I20260810 19:02:39.076047 23507 tablet_replica.cc:330] T 75593be963b34cf08b6d7782905e4b4b P 92166c3154a342618227e85623ad57c8: stopping tablet replica
I20260810 19:02:39.076275 23508 tablet_replica.cc:330] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: stopping tablet replica
I20260810 19:02:39.076587 23508 raft_consensus.cc:2229] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:39.076655 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 22773
I20260810 19:02:39.084030 23508 raft_consensus.cc:2258] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:39.086036 23510 tablet_replica.cc:330] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: stopping tablet replica
I20260810 19:02:39.087220 23508 ts_tablet_manager.cc:1905] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:39.087357 23510 raft_consensus.cc:2229] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:39.087714 23510 raft_consensus.cc:2258] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:39.089265 23510 ts_tablet_manager.cc:1905] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:39.090308 23508 ts_tablet_manager.cc:1918] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:39.090416 23508 log.cc:1196] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/75593be963b34cf08b6d7782905e4b4b
I20260810 19:02:39.090878 23508 ts_tablet_manager.cc:1939] T 75593be963b34cf08b6d7782905e4b4b P f11737f061274b458b50ac70b893502c: Deleting consensus metadata
I20260810 19:02:39.092048 23510 ts_tablet_manager.cc:1918] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:39.092149 23510 log.cc:1196] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/75593be963b34cf08b6d7782905e4b4b
I20260810 19:02:39.092229 22656 catalog_manager.cc:4841] TS f11737f061274b458b50ac70b893502c (127.27.147.131:42377): tablet 75593be963b34cf08b6d7782905e4b4b (table test-table [id=f07c55d22997416893b07ae8b380ed2c]) successfully deleted
I20260810 19:02:39.092466 23510 ts_tablet_manager.cc:1939] T 75593be963b34cf08b6d7782905e4b4b P a4ce95e7467541fc93f4ee29543e6bcf: Deleting consensus metadata
I20260810 19:02:39.093470 22656 catalog_manager.cc:4841] TS a4ce95e7467541fc93f4ee29543e6bcf (127.27.147.130:40183): tablet 75593be963b34cf08b6d7782905e4b4b (table test-table [id=f07c55d22997416893b07ae8b380ed2c]) successfully deleted
W20260810 19:02:39.094544 22653 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:38441 (error 108)
I20260810 19:02:39.095352 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 22998
I20260810 19:02:39.095664 23465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.095313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:38441 (local address 127.0.0.1:45978)
0810 19:02:39.095490 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:39.095566 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:38441: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 19:02:39.101303 22653 catalog_manager.cc:4584] TS 92166c3154a342618227e85623ad57c8 (127.27.147.129:38441): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 75593be963b34cf08b6d7782905e4b4b: Network error: Client connection negotiation failed: client connection to 127.27.147.129:38441: connect: Connection refused (error 111)
I20260810 19:02:39.110798 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 23229
I20260810 19:02:39.132503 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 22626
2026-08-10T19:02:39Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2773 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T19:02:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:39Z Disabled control of system clock
I20260810 19:02:39.167809 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37529
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:02:39.279623 23518 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:39.279850 23518 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:39.279883 23518 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:39.283779 23518 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:39.283828 23518 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:39.283843 23518 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:39.283859 23518 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:39.283874 23518 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:39.288379 23518 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37529
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23518
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:39.289259 23518 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:39.290988 23518 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:39.298960 23528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:39.298986 23531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:39.299377 23533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:39.299377 23518 server_base.cc:1034] running on GCE node
I20260810 19:02:39.300185 23518 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:39.301470 23518 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:39.302665 23518 hybrid_clock.cc:648] HybridClock initialized: now 1786388559302649 us; error 33 us; skew 500 ppm
I20260810 19:02:39.304548 23518 webserver.cc:460] Webserver started at http://127.27.147.190:36375/ using document root <none> and password file <none>
I20260810 19:02:39.305115 23518 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:39.305177 23518 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:39.305413 23518 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:39.307497 23518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "1a54f600a7d34b688a80f962d671841f"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "b570fc77d1853328a0d246afed8c0738"
server_key_iv: "22de5b98ecea6ce5a2ed07205cb9ebf6"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.308060 23518 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "1a54f600a7d34b688a80f962d671841f"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "b570fc77d1853328a0d246afed8c0738"
server_key_iv: "22de5b98ecea6ce5a2ed07205cb9ebf6"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.311607 23518 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:02:39.313930 23543 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.315160 23518 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:02:39.315299 23518 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "1a54f600a7d34b688a80f962d671841f"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "b570fc77d1853328a0d246afed8c0738"
server_key_iv: "22de5b98ecea6ce5a2ed07205cb9ebf6"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.315407 23518 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:39.331094 23518 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:39.331704 23518 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:39.331888 23518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:39.338919 23642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37529 every 8 connection(s)
I20260810 19:02:39.338961 23518 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37529
I20260810 19:02:39.339821 23518 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:39.342787 23643 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:39.343478 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 23518
I20260810 19:02:39.343627 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:39.343905 28238 external_mini_cluster.cc:1428] Setting key 9f5ad65dfbaf19028af86c85c7a62d12
I20260810 19:02:39.348578 23643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f: Bootstrap starting.
I20260810 19:02:39.350728 23643 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:39.351475 23643 log.cc:824] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:39.352681 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.345262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37040 (local address 127.27.147.190:37529)
0810 19:02:39.345861 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:39.345875 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:39.345907 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:02:39.346548 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:39.346565 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:39.346995 (+ 430us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:39.347309 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:39.347725 (+ 416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.348638 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.349570 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.350028 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.350201 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.351101 (+ 900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:39.351106 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:39.351118 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:39.351466 (+ 348us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:39.351516 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:39.351535 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:39.351645 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:39.351789 (+ 144us) server_negotiation.cc:299] Negotiation successful
0810 19:02:39.352055 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":128,"threads_started":1}
I20260810 19:02:39.353241 23643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f: No bootstrap required, opened a new log
I20260810 19:02:39.356846 23643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a54f600a7d34b688a80f962d671841f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37529 } }
I20260810 19:02:39.357066 23643 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:39.357142 23643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a54f600a7d34b688a80f962d671841f, State: Initialized, Role: FOLLOWER
I20260810 19:02:39.357577 23643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a54f600a7d34b688a80f962d671841f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37529 } }
I20260810 19:02:39.357723 23643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:39.357808 23643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:39.357906 23643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:39.358868 23643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a54f600a7d34b688a80f962d671841f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37529 } }
I20260810 19:02:39.359197 23643 leader_election.cc:302] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a54f600a7d34b688a80f962d671841f; no voters:
I20260810 19:02:39.359483 23643 leader_election.cc:288] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:39.359606 23650 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:39.359871 23650 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [term 1 LEADER]: Becoming Leader. State: Replica: 1a54f600a7d34b688a80f962d671841f, State: Running, Role: LEADER
I20260810 19:02:39.360215 23650 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a54f600a7d34b688a80f962d671841f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37529 } }
I20260810 19:02:39.360567 23643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:39.361982 23654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a54f600a7d34b688a80f962d671841f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a54f600a7d34b688a80f962d671841f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37529 } } }
I20260810 19:02:39.361968 23655 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a54f600a7d34b688a80f962d671841f. Latest consensus state: current_term: 1 leader_uuid: "1a54f600a7d34b688a80f962d671841f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a54f600a7d34b688a80f962d671841f" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37529 } } }
I20260810 19:02:39.362118 23654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:39.362118 23655 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:39.362555 23666 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:39.365182 23666 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:39.369949 23666 catalog_manager.cc:1347] Generated new cluster ID: 9b069b15237c429b93892a690327f34e
I20260810 19:02:39.370012 23666 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:39.387111 23666 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:39.387914 23666 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:39.400681 23666 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f: Generated new TSK 0
I20260810 19:02:39.401278 23666 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:39.404727 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37529
--builtin_ntp_servers=127.27.147.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:02:39.517999 23682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:39.518324 23682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:39.518410 23682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:39.522390 23682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:39.522466 23682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:39.522553 23682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:02:39.527658 23682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:39.528672 23682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:39.529978 23682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:39.539412 23703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:39.539482 23688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:39.539572 23682 server_base.cc:1034] running on GCE node
W20260810 19:02:39.539413 23696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:39.540123 23682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:39.540725 23682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:39.541916 23682 hybrid_clock.cc:648] HybridClock initialized: now 1786388559541902 us; error 28 us; skew 500 ppm
I20260810 19:02:39.543704 23682 webserver.cc:460] Webserver started at http://127.27.147.129:32801/ using document root <none> and password file <none>
I20260810 19:02:39.544269 23682 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:39.544363 23682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:39.544601 23682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:39.546545 23682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "8b378f8a84e840058fa90cc374b8532e"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "b8b2e7acc9fd09562ed995a12d513b2e"
server_key_iv: "f432a03932ac8274b10dedc27f06daba"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.547139 23682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "8b378f8a84e840058fa90cc374b8532e"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "b8b2e7acc9fd09562ed995a12d513b2e"
server_key_iv: "f432a03932ac8274b10dedc27f06daba"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.550590 23682 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:02:39.553023 23712 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.554168 23682 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:39.554304 23682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "8b378f8a84e840058fa90cc374b8532e"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "b8b2e7acc9fd09562ed995a12d513b2e"
server_key_iv: "f432a03932ac8274b10dedc27f06daba"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.554414 23682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:39.572288 23682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:39.572880 23682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:39.573072 23682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:39.573616 23682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:39.574513 23682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:39.574587 23682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.574653 23682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:39.574703 23682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.583945 23682 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:40915
I20260810 19:02:39.584010 23895 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:40915 every 8 connection(s)
I20260810 19:02:39.584815 23682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:02:39.591986 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 23682
I20260810 19:02:39.592087 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:02:39.592391 28238 external_mini_cluster.cc:1428] Setting key 9298cd86e3d7237c04f3bf8b077b1104
I20260810 19:02:39.593463 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.586572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:60063 (local address 127.27.147.190:37529)
0810 19:02:39.586702 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:39.586706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:39.587721 (+ 1015us) server_negotiation.cc:407] Connection header received
0810 19:02:39.589022 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:39.589024 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:39.589076 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:39.589150 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:39.589897 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.590427 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.591501 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.591757 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.591874 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.592990 (+ 1116us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:39.592994 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:39.592997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:39.593009 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:39.593030 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:39.593040 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:39.593121 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:39.593297 (+ 176us) server_negotiation.cc:299] Negotiation successful
0810 19:02:39.593351 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:02:39.594369 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.586993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37529 (local address 127.27.147.129:60063)
0810 19:02:39.587608 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:39.587644 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:39.588842 (+ 1198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:39.589264 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:39.589269 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:39.589344 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:39.589781 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:39.589792 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.590535 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.590537 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:39.591397 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:39.591407 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.591901 (+ 494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.591904 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:39.592033 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.592678 (+ 645us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:39.592680 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:39.592716 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:39.592809 (+ 93us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:39.593147 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:39.593157 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:39.593163 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 19:02:39.593290 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 19:02:39.593559 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"thread_start_us":119,"threads_started":1}
I20260810 19:02:39.594468 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37529
--builtin_ntp_servers=127.27.147.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:02:39.595583 23896 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37529
I20260810 19:02:39.595815 23896 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:39.596282 23896 heartbeater.cc:511] Master 127.27.147.190:37529 requested a full tablet report, sending...
I20260810 19:02:39.597699 23574 ts_manager.cc:194] Registered new tserver with Master: 8b378f8a84e840058fa90cc374b8532e (127.27.147.129:40915)
I20260810 19:02:39.598877 23574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.129:60063
W20260810 19:02:39.706173 23899 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:39.706396 23899 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:39.706470 23899 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:39.710381 23899 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:39.710433 23899 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:39.710515 23899 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:02:39.715164 23899 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23899
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:39.716037 23899 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:39.717258 23899 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:02:39.726903 23899 server_base.cc:1034] running on GCE node
W20260810 19:02:39.726835 23906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:39.726856 23905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:39.726871 23909 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:39.727540 23899 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:39.728224 23899 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:39.729440 23899 hybrid_clock.cc:648] HybridClock initialized: now 1786388559729420 us; error 37 us; skew 500 ppm
I20260810 19:02:39.731153 23899 webserver.cc:460] Webserver started at http://127.27.147.130:34595/ using document root <none> and password file <none>
I20260810 19:02:39.731657 23899 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:39.731716 23899 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:39.731870 23899 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:39.733711 23899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "93e67def742c49dda93240aa97e4dca1"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "857343fb1d0751cc8a919c2d1dd6a85f"
server_key_iv: "4addd42659ade6efa930af43f2aa5907"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.734205 23899 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "93e67def742c49dda93240aa97e4dca1"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "857343fb1d0751cc8a919c2d1dd6a85f"
server_key_iv: "4addd42659ade6efa930af43f2aa5907"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.737833 23899 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 19:02:39.740118 23917 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.741289 23899 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:02:39.741380 23899 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "93e67def742c49dda93240aa97e4dca1"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "857343fb1d0751cc8a919c2d1dd6a85f"
server_key_iv: "4addd42659ade6efa930af43f2aa5907"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.741466 23899 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:39.760931 23899 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:39.761659 23899 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:39.761862 23899 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:39.763576 23899 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:39.764508 23899 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:39.764581 23899 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.764654 23899 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:39.764706 23899 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.773227 23899 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:39991
I20260810 19:02:39.773260 24100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:39991 every 8 connection(s)
I20260810 19:02:39.774098 23899 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:02:39.781773 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 23899
I20260810 19:02:39.781909 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:02:39.782220 28238 external_mini_cluster.cc:1428] Setting key af5969d1372d7be6a0bbb60737fc8275
I20260810 19:02:39.782917 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.775927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:51579 (local address 127.27.147.190:37529)
0810 19:02:39.776080 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:39.776084 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:39.777065 (+ 981us) server_negotiation.cc:407] Connection header received
0810 19:02:39.778392 (+ 1327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:39.778395 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:39.778439 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:39.778510 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:39.779399 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.779920 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.780998 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.781205 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.781280 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.782378 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:39.782381 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:39.782384 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:39.782398 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:39.782423 (+ 25us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:39.782432 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:39.782516 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:39.782676 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 19:02:39.782740 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:02:39.783788 24106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.776283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37529 (local address 127.27.147.130:51579)
0810 19:02:39.776926 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:39.776966 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:39.778206 (+ 1240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:39.778650 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:39.778657 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:39.778732 (+ 75us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:39.779267 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:39.779280 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.780031 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.780033 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:39.780890 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:39.780900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.781332 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.781334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:39.781477 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.782109 (+ 632us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:39.782111 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:39.782148 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:39.782207 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:39.782520 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:39.782532 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:39.782540 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 19:02:39.782674 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 19:02:39.782953 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":113,"threads_started":1}
I20260810 19:02:39.784258 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37529
--builtin_ntp_servers=127.27.147.148:32983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:02:39.784900 24102 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37529
I20260810 19:02:39.785135 24102 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:39.785701 24102 heartbeater.cc:511] Master 127.27.147.190:37529 requested a full tablet report, sending...
I20260810 19:02:39.786850 23574 ts_manager.cc:194] Registered new tserver with Master: 93e67def742c49dda93240aa97e4dca1 (127.27.147.130:39991)
I20260810 19:02:39.787424 23574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.130:51579
W20260810 19:02:39.895470 24108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:39.895738 24108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:39.895805 24108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:39.899883 24108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:39.899986 24108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:39.900064 24108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:39.904702 24108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:32983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:37529
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24108
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:39.905727 24108 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:39.907109 24108 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:39.916627 24128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:39.916590 24118 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:39.916607 24115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:39.916813 24108 server_base.cc:1034] running on GCE node
I20260810 19:02:39.917326 24108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:39.917951 24108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:39.919140 24108 hybrid_clock.cc:648] HybridClock initialized: now 1786388559919123 us; error 32 us; skew 500 ppm
I20260810 19:02:39.921037 24108 webserver.cc:460] Webserver started at http://127.27.147.131:46495/ using document root <none> and password file <none>
I20260810 19:02:39.921622 24108 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:39.921720 24108 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:39.921968 24108 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:39.924160 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "6113574f19573e984aa77436dc5414cf"
server_key_iv: "a48dde7a092c96c1256612a95032afc6"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.924723 24108 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "6113574f19573e984aa77436dc5414cf"
server_key_iv: "a48dde7a092c96c1256612a95032afc6"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.928463 24108 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:02:39.931020 24137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.932305 24108 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:39.932471 24108 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39"
format_stamp: "Formatted at 2026-08-10 19:02:39 on dist-test-slave-znh6"
server_key: "6113574f19573e984aa77436dc5414cf"
server_key_iv: "a48dde7a092c96c1256612a95032afc6"
server_key_version: "encryptionkey@0"
I20260810 19:02:39.932587 24108 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:39.951819 24108 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:39.952472 24108 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:39.952673 24108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:39.953279 24108 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:39.954252 24108 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:39.954304 24108 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.954394 24108 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:39.954437 24108 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:39.964885 24108 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:40809
I20260810 19:02:39.964917 24305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:40809 every 8 connection(s)
I20260810 19:02:39.965828 24108 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:02:39.971633 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 24108
I20260810 19:02:39.971730 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:02:39.972036 28238 external_mini_cluster.cc:1428] Setting key 4b397d65337d14b2608d5e1cf67e3ee5
I20260810 19:02:39.975737 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.967726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:35183 (local address 127.27.147.190:37529)
0810 19:02:39.967880 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:39.967885 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:39.968952 (+ 1067us) server_negotiation.cc:407] Connection header received
0810 19:02:39.970374 (+ 1422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:39.970378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:39.970449 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:39.970560 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:39.971484 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.972117 (+ 633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.973526 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.973913 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.974038 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.975291 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:39.975295 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:39.975297 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:39.975310 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:39.975332 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:39.975341 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:39.975410 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:39.975556 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 19:02:39.975613 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 19:02:39.976578 24314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.968135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:37529 (local address 127.27.147.131:35183)
0810 19:02:39.968800 (+ 665us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:39.968839 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:39.970154 (+ 1315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:39.970741 (+ 587us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:39.970748 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:39.970846 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:39.971317 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:39.971330 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.972269 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.972272 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:39.973376 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:39.973387 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.974098 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.974102 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:39.974321 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.975029 (+ 708us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:39.975031 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:39.975070 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:39.975132 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:39.975422 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:39.975437 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:39.975441 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 19:02:39.975536 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 19:02:39.975760 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":136,"threads_started":1}
I20260810 19:02:39.977669 24306 heartbeater.cc:347] Connected to a master server at 127.27.147.190:37529
I20260810 19:02:39.977933 24306 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:39.978441 24306 heartbeater.cc:511] Master 127.27.147.190:37529 requested a full tablet report, sending...
I20260810 19:02:39.979514 23574 ts_manager.cc:194] Registered new tserver with Master: 0d431e6a4e464ac1ad4c87a0ee3f9c39 (127.27.147.131:40809)
I20260810 19:02:39.980141 23574 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.131:35183
I20260810 19:02:39.986124 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:02:39.991727 23648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:39.988116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37048 (local address 127.27.147.190:37529)
0810 19:02:39.988257 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:39.988261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:39.988499 (+ 238us) server_negotiation.cc:407] Connection header received
0810 19:02:39.988591 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:39.988593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:39.988631 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:39.988695 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:39.988986 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.989484 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.990301 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:39.990508 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:39.990580 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:39.991296 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:39.991299 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:39.991301 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:39.991312 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:39.991328 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:39.991336 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:39.991393 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:39.991497 (+ 104us) server_negotiation.cc:299] Negotiation successful
0810 19:02:39.991542 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:02:39.994251 23574 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37048:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:02:39.997244 23574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:02:40.011751 24327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.005636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:40809 (local address 127.0.0.1:52392)
0810 19:02:40.006473 (+ 837us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:40.006522 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:40.006641 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:40.007451 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:40.007457 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:40.007481 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:40.007623 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.007630 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.008854 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.008859 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.010082 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.010093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.011194 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.011198 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.011317 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.011362 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 19:02:40.011483 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 19:02:40.011609 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":112,"threads_started":1}
I20260810 19:02:40.011819 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.006450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52392 (local address 127.27.147.131:40809)
0810 19:02:40.006916 (+ 466us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:40.006938 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:40.006964 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:02:40.007043 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:40.007048 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:40.007277 (+ 229us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:40.007439 (+ 162us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:40.007760 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.008747 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.010223 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.011058 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.011178 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.011343 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:40.011533 (+ 190us) server_negotiation.cc:299] Negotiation successful
0810 19:02:40.011635 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":329,"thread_start_us":151,"threads_started":1}
I20260810 19:02:40.012421 24332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.005650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:40915 (local address 127.0.0.1:33850)
0810 19:02:40.006784 (+ 1134us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:40.006798 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:40.006925 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:40.008179 (+ 1254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:40.008182 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:40.008199 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:40.008285 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.008290 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.009542 (+ 1252us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.009547 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.010664 (+ 1117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.010676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.011998 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.012003 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.012104 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.012123 (+ 19us) client_negotiation.cc:769] Sending connection context
0810 19:02:40.012215 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 19:02:40.012290 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1065,"mutex_wait_us":1,"spinlock_wait_cycles":6400,"thread_start_us":100,"threads_started":1}
I20260810 19:02:40.012564 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.005797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33850 (local address 127.27.147.129:40915)
0810 19:02:40.007548 (+ 1751us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:40.007561 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:40.007585 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:02:40.007664 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:40.007672 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:40.007880 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:40.008035 (+ 155us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:40.008427 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.009421 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.010835 (+ 1414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.011636 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.011757 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.011927 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:40.012265 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 19:02:40.012384 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1646,"thread_start_us":119,"threads_started":1}
I20260810 19:02:40.012224 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.005939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:39991 (local address 127.0.0.1:33720)
0810 19:02:40.007204 (+ 1265us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:40.007216 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:40.007295 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:40.007744 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:40.007747 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:40.007766 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:40.007870 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.007879 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.009698 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.009702 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.010616 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.010625 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.011828 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.011833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.011947 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.011969 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 19:02:40.012035 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 19:02:40.012095 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1198,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260810 19:02:40.013669 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.006357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.27.147.130:39991)
0810 19:02:40.007167 (+ 810us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:40.007178 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:40.007312 (+ 134us) server_negotiation.cc:407] Connection header received
0810 19:02:40.007398 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:40.007403 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:40.007591 (+ 188us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:40.007761 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:40.008009 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.009026 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.010754 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.011481 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.011591 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.011754 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:40.013466 (+ 1712us) server_negotiation.cc:299] Negotiation successful
0810 19:02:40.013545 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":712,"thread_start_us":149,"threads_started":1}
I20260810 19:02:40.015486 23794 tablet_service.cc:1467] Processing CreateTablet for tablet dce3267cb48c460d825f3ee6db6f05b4 (DEFAULT_TABLE table=test-table [id=b072db734f2b4802bfc864be9e1e4a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:40.015486 24204 tablet_service.cc:1467] Processing CreateTablet for tablet dce3267cb48c460d825f3ee6db6f05b4 (DEFAULT_TABLE table=test-table [id=b072db734f2b4802bfc864be9e1e4a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:40.015961 23999 tablet_service.cc:1467] Processing CreateTablet for tablet dce3267cb48c460d825f3ee6db6f05b4 (DEFAULT_TABLE table=test-table [id=b072db734f2b4802bfc864be9e1e4a5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:40.016656 23794 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce3267cb48c460d825f3ee6db6f05b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:40.016656 24204 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce3267cb48c460d825f3ee6db6f05b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:40.017086 23999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dce3267cb48c460d825f3ee6db6f05b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:40.023420 24338 tablet_bootstrap.cc:492] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Bootstrap starting.
I20260810 19:02:40.024828 24340 tablet_bootstrap.cc:492] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: Bootstrap starting.
I20260810 19:02:40.024850 24339 tablet_bootstrap.cc:492] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Bootstrap starting.
I20260810 19:02:40.025593 24338 tablet_bootstrap.cc:654] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:40.026333 24338 log.cc:824] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:40.027070 24340 tablet_bootstrap.cc:654] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:40.027086 24339 tablet_bootstrap.cc:654] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:40.027935 24339 log.cc:824] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:40.027946 24340 log.cc:824] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:40.028429 24338 tablet_bootstrap.cc:492] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: No bootstrap required, opened a new log
I20260810 19:02:40.028631 24338 ts_tablet_manager.cc:1397] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:02:40.029922 24339 tablet_bootstrap.cc:492] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: No bootstrap required, opened a new log
I20260810 19:02:40.030118 24339 ts_tablet_manager.cc:1397] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 19:02:40.030507 24340 tablet_bootstrap.cc:492] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: No bootstrap required, opened a new log
I20260810 19:02:40.030694 24340 ts_tablet_manager.cc:1397] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 19:02:40.031828 24338 raft_consensus.cc:348] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.032012 24338 raft_consensus.cc:374] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:40.032043 24338 raft_consensus.cc:729] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93e67def742c49dda93240aa97e4dca1, State: Initialized, Role: FOLLOWER
I20260810 19:02:40.032477 24338 consensus_queue.cc:260] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.033212 24102 heartbeater.cc:503] Master 127.27.147.190:37529 was elected leader, sending a full tablet report...
I20260810 19:02:40.033493 24338 ts_tablet_manager.cc:1428] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260810 19:02:40.034410 24339 raft_consensus.cc:348] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.034410 24340 raft_consensus.cc:348] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.034669 24339 raft_consensus.cc:374] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:40.034669 24340 raft_consensus.cc:374] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:40.034729 24340 raft_consensus.cc:729] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b378f8a84e840058fa90cc374b8532e, State: Initialized, Role: FOLLOWER
I20260810 19:02:40.034771 24339 raft_consensus.cc:729] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d431e6a4e464ac1ad4c87a0ee3f9c39, State: Initialized, Role: FOLLOWER
I20260810 19:02:40.035202 24339 consensus_queue.cc:260] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.035202 24340 consensus_queue.cc:260] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.036149 24339 ts_tablet_manager.cc:1428] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 19:02:40.036149 24340 ts_tablet_manager.cc:1428] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: Time spent starting tablet: real 0.005s user 0.007s sys 0.000s
I20260810 19:02:40.036161 23896 heartbeater.cc:503] Master 127.27.147.190:37529 was elected leader, sending a full tablet report...
I20260810 19:02:40.036516 24306 heartbeater.cc:503] Master 127.27.147.190:37529 was elected leader, sending a full tablet report...
W20260810 19:02:40.085916 23897 tablet.cc:2369] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:02:40.145586 24350 raft_consensus.cc:482] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:40.145810 24350 raft_consensus.cc:504] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.147040 24350 leader_election.cc:288] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d431e6a4e464ac1ad4c87a0ee3f9c39 (127.27.147.131:40809), 93e67def742c49dda93240aa97e4dca1 (127.27.147.130:39991)
I20260810 19:02:40.150990 24334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.147423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:58859 (local address 127.27.147.130:39991)
0810 19:02:40.147579 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:40.147585 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:40.147885 (+ 300us) server_negotiation.cc:407] Connection header received
0810 19:02:40.148017 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:40.148020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:40.148069 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:40.148149 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:40.148443 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.148914 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.150024 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.150378 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.150470 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.150535 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:40.150755 (+ 220us) server_negotiation.cc:299] Negotiation successful
0810 19:02:40.150854 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:02:40.150993 24355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.147273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:39991 (local address 127.27.147.129:58859)
0810 19:02:40.147752 (+ 479us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:40.147763 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:40.147895 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:40.148178 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:40.148180 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:40.148195 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:40.148321 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.148329 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.149067 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.149070 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.149876 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.149885 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.150500 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.150506 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.150611 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.150640 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 19:02:40.150709 (+ 69us) client_negotiation.cc:240] Negotiation successful
0810 19:02:40.150799 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":23,"thread_start_us":81,"threads_started":1}
I20260810 19:02:40.151013 24335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.147306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:55603 (local address 127.27.147.131:40809)
0810 19:02:40.147439 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:40.147444 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:40.147688 (+ 244us) server_negotiation.cc:407] Connection header received
0810 19:02:40.147867 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:40.147871 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:40.147919 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:40.148000 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:40.148332 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.148879 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.149981 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.150446 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.150560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.150607 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:40.150836 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 19:02:40.150913 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:02:40.151011 24356 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.147246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:40809 (local address 127.27.147.129:55603)
0810 19:02:40.147593 (+ 347us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:40.147605 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:40.147733 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:40.148021 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:40.148024 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:40.148043 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:40.148201 (+ 158us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.148209 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.148993 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.149000 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.149853 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:40.149863 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.150577 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.150580 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:40.150665 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.150683 (+ 18us) client_negotiation.cc:769] Sending connection context
0810 19:02:40.150780 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 19:02:40.150864 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":20,"thread_start_us":107,"threads_started":1}
I20260810 19:02:40.151695 24029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce3267cb48c460d825f3ee6db6f05b4" candidate_uuid: "8b378f8a84e840058fa90cc374b8532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e67def742c49dda93240aa97e4dca1" is_pre_election: true
I20260810 19:02:40.151857 24236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce3267cb48c460d825f3ee6db6f05b4" candidate_uuid: "8b378f8a84e840058fa90cc374b8532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" is_pre_election: true
I20260810 19:02:40.151999 24029 raft_consensus.cc:2454] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b378f8a84e840058fa90cc374b8532e in term 0.
I20260810 19:02:40.152207 24236 raft_consensus.cc:2454] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8b378f8a84e840058fa90cc374b8532e in term 0.
I20260810 19:02:40.152575 23713 leader_election.cc:302] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8b378f8a84e840058fa90cc374b8532e, 93e67def742c49dda93240aa97e4dca1; no voters:
I20260810 19:02:40.152910 24350 raft_consensus.cc:2781] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:02:40.153081 24350 raft_consensus.cc:482] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:02:40.153159 24350 raft_consensus.cc:3037] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:40.154184 24350 raft_consensus.cc:504] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.154548 24350 leader_election.cc:288] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [CANDIDATE]: Term 1 election: Requested vote from peers 0d431e6a4e464ac1ad4c87a0ee3f9c39 (127.27.147.131:40809), 93e67def742c49dda93240aa97e4dca1 (127.27.147.130:39991)
I20260810 19:02:40.154879 24236 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce3267cb48c460d825f3ee6db6f05b4" candidate_uuid: "8b378f8a84e840058fa90cc374b8532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39"
I20260810 19:02:40.155007 24236 raft_consensus.cc:3037] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:40.155032 24029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dce3267cb48c460d825f3ee6db6f05b4" candidate_uuid: "8b378f8a84e840058fa90cc374b8532e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93e67def742c49dda93240aa97e4dca1"
I20260810 19:02:40.155172 24029 raft_consensus.cc:3037] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:40.156102 24236 raft_consensus.cc:2454] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b378f8a84e840058fa90cc374b8532e in term 1.
I20260810 19:02:40.156368 24029 raft_consensus.cc:2454] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8b378f8a84e840058fa90cc374b8532e in term 1.
I20260810 19:02:40.156522 23715 leader_election.cc:302] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d431e6a4e464ac1ad4c87a0ee3f9c39, 8b378f8a84e840058fa90cc374b8532e; no voters:
I20260810 19:02:40.156704 24350 raft_consensus.cc:2781] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:40.156941 24350 raft_consensus.cc:686] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [term 1 LEADER]: Becoming Leader. State: Replica: 8b378f8a84e840058fa90cc374b8532e, State: Running, Role: LEADER
I20260810 19:02:40.157296 24350 consensus_queue.cc:237] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } }
I20260810 19:02:40.160567 23573 catalog_manager.cc:5495] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8b378f8a84e840058fa90cc374b8532e (127.27.147.129). New cstate: current_term: 1 leader_uuid: "8b378f8a84e840058fa90cc374b8532e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8b378f8a84e840058fa90cc374b8532e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 40915 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 } health_report { overall_health: UNKNOWN } } }
I20260810 19:02:40.197490 24329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.194220 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33860 (local address 127.27.147.129:40915)
0810 19:02:40.194342 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:40.194345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:40.194358 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:02:40.194502 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:40.194505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:40.194561 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:40.194653 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:40.194908 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.195394 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.196239 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:40.196455 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:40.196528 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:40.196789 (+ 261us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:40.197183 (+ 394us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:02:40.197272 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:40.197328 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 19:02:40.197373 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:02:40.203263 24029 raft_consensus.cc:1261] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 1 FOLLOWER]: Refusing update from remote peer 8b378f8a84e840058fa90cc374b8532e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:40.203261 24236 raft_consensus.cc:1261] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 1 FOLLOWER]: Refusing update from remote peer 8b378f8a84e840058fa90cc374b8532e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:40.203996 24350 consensus_queue.cc:1035] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [LEADER]: Connected to new peer: Peer: permanent_uuid: "93e67def742c49dda93240aa97e4dca1" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39991 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:40.204214 24357 consensus_queue.cc:1035] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d431e6a4e464ac1ad4c87a0ee3f9c39" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40809 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:40.211612 24363 mvcc.cc:205] Tried to move back new op lower bound from 7317047542585049088 to 7317047542407069696. Current Snapshot: MvccSnapshot[applied={T|T < 7317047542585049088}]
I20260810 19:02:40.220845 23573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:40.221010 23573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37048:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:40.223338 23573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a54f600a7d34b688a80f962d671841f: Sending DeleteTablet for 3 replicas of tablet dce3267cb48c460d825f3ee6db6f05b4
I20260810 19:02:40.224079 24204 tablet_service.cc:1514] Processing DeleteTablet for tablet dce3267cb48c460d825f3ee6db6f05b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:40 UTC) from {username='slave'} at 127.0.0.1:52392
I20260810 19:02:40.224095 23794 tablet_service.cc:1514] Processing DeleteTablet for tablet dce3267cb48c460d825f3ee6db6f05b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:40 UTC) from {username='slave'} at 127.0.0.1:33850
I20260810 19:02:40.224210 23999 tablet_service.cc:1514] Processing DeleteTablet for tablet dce3267cb48c460d825f3ee6db6f05b4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:40 UTC) from {username='slave'} at 127.0.0.1:33720
I20260810 19:02:40.224620 24380 tablet_replica.cc:330] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: stopping tablet replica
I20260810 19:02:40.224692 24377 tablet_replica.cc:330] T dce3267cb48c460d825f3ee6db6f05b4 P 8b378f8a84e840058fa90cc374b8532e: stopping tablet replica
I20260810 19:02:40.224993 24380 raft_consensus.cc:2229] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:40.225180 24379 tablet_replica.cc:330] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: stopping tablet replica
I20260810 19:02:40.225272 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 23682
I20260810 19:02:40.228020 24380 raft_consensus.cc:2258] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:40.228494 24379 raft_consensus.cc:2229] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:40.228786 24379 raft_consensus.cc:2258] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:40.230125 24380 ts_tablet_manager.cc:1905] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:40.233040 24379 ts_tablet_manager.cc:1905] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:40.233927 24380 ts_tablet_manager.cc:1918] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:40.234040 24380 log.cc:1196] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/dce3267cb48c460d825f3ee6db6f05b4
I20260810 19:02:40.234427 24380 ts_tablet_manager.cc:1939] T dce3267cb48c460d825f3ee6db6f05b4 P 93e67def742c49dda93240aa97e4dca1: Deleting consensus metadata
I20260810 19:02:40.236025 24379 ts_tablet_manager.cc:1918] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:40.236107 24379 log.cc:1196] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/dce3267cb48c460d825f3ee6db6f05b4
I20260810 19:02:40.236078 23544 catalog_manager.cc:4841] TS 93e67def742c49dda93240aa97e4dca1 (127.27.147.130:39991): tablet dce3267cb48c460d825f3ee6db6f05b4 (table test-table [id=b072db734f2b4802bfc864be9e1e4a5a]) successfully deleted
I20260810 19:02:40.236377 24379 ts_tablet_manager.cc:1939] T dce3267cb48c460d825f3ee6db6f05b4 P 0d431e6a4e464ac1ad4c87a0ee3f9c39: Deleting consensus metadata
I20260810 19:02:40.244072 23547 catalog_manager.cc:4841] TS 0d431e6a4e464ac1ad4c87a0ee3f9c39 (127.27.147.131:40809): tablet dce3267cb48c460d825f3ee6db6f05b4 (table test-table [id=b072db734f2b4802bfc864be9e1e4a5a]) successfully deleted
W20260810 19:02:40.250947 23548 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:40915 (error 108)
I20260810 19:02:40.251858 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 23899
I20260810 19:02:40.252146 24326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:40.251786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:40915 (local address 127.0.0.1:33876)
0810 19:02:40.251936 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:40.252044 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:40915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260810 19:02:40.254869 23548 catalog_manager.cc:4584] TS 8b378f8a84e840058fa90cc374b8532e (127.27.147.129:40915): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet dce3267cb48c460d825f3ee6db6f05b4: Network error: Client connection negotiation failed: client connection to 127.27.147.129:40915: connect: Connection refused (error 111)
I20260810 19:02:40.264174 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 24108
I20260810 19:02:40.275542 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 23518
2026-08-10T19:02:40Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (1142 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:40 dist-test-slave-znh6 krb5kdc[24389](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:40 dist-test-slave-znh6 krb5kdc[24389](info): set up 2 sockets
Aug 10 19:02:40 dist-test-slave-znh6 krb5kdc[24389](info): commencing operation
krb5kdc: starting...
W20260810 19:02:41.538961 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.237s user 0.000s sys 0.008s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:41 dist-test-slave-znh6 krb5kdc[24389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:02:41.663643 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35609
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:35609
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:41.773736 24410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:41.774008 24410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:41.774076 24410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:41.778030 24410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:41.778116 24410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:41.778155 24410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:41.778191 24410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:41.778225 24410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:41.783293 24410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:35609
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24410
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:41.784240 24410 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:41.785185 24410 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:02:41.793790 24410 server_base.cc:1034] running on GCE node
W20260810 19:02:41.793979 24418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:41.793992 24423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:41.793968 24421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:41.794710 24410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:41.795761 24410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:41.796962 24410 hybrid_clock.cc:648] HybridClock initialized: now 1786388561796942 us; error 31 us; skew 500 ppm
Aug 10 19:02:41 dist-test-slave-znh6 krb5kdc[24389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388561, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:41.804349 24410 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:02:41.805148 24410 webserver.cc:460] Webserver started at http://127.27.147.190:37537/ using document root <none> and password file <none>
I20260810 19:02:41.805639 24410 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:41.805696 24410 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:41.805861 24410 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:41.807837 24410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "49c34f294276406bbc2e6df08beba708"
format_stamp: "Formatted at 2026-08-10 19:02:41 on dist-test-slave-znh6"
server_key: "2aa37fd15b120e72ee8a5ca7e4285207"
server_key_iv: "709ad4d9bf61dd91149fb62a5325ec09"
server_key_version: "encryptionkey@0"
I20260810 19:02:41.808337 24410 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "49c34f294276406bbc2e6df08beba708"
format_stamp: "Formatted at 2026-08-10 19:02:41 on dist-test-slave-znh6"
server_key: "2aa37fd15b120e72ee8a5ca7e4285207"
server_key_iv: "709ad4d9bf61dd91149fb62a5325ec09"
server_key_version: "encryptionkey@0"
I20260810 19:02:41.811853 24410 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:02:41.814164 24434 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:41.815454 24410 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:02:41.815583 24410 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "49c34f294276406bbc2e6df08beba708"
format_stamp: "Formatted at 2026-08-10 19:02:41 on dist-test-slave-znh6"
server_key: "2aa37fd15b120e72ee8a5ca7e4285207"
server_key_iv: "709ad4d9bf61dd91149fb62a5325ec09"
server_key_version: "encryptionkey@0"
I20260810 19:02:41.815688 24410 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:41.839457 24410 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:41.842170 24410 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:41.842373 24410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:41.849514 24410 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:35609
I20260810 19:02:41.849617 24510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:35609 every 8 connection(s)
I20260810 19:02:41.850427 24410 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:41.853214 24511 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:41.858415 24511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708: Bootstrap starting.
I20260810 19:02:41.859474 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 24410
I20260810 19:02:41.859599 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:41.859946 28238 external_mini_cluster.cc:1428] Setting key 008955fb71382458c4a0768dce02782d
I20260810 19:02:41.861148 24511 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:41.861892 24511 log.cc:824] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:41.864166 24511 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708: No bootstrap required, opened a new log
I20260810 19:02:41.868499 24511 raft_consensus.cc:348] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c34f294276406bbc2e6df08beba708" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35609 } }
I20260810 19:02:41.868798 24511 raft_consensus.cc:374] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:41.868882 24511 raft_consensus.cc:729] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c34f294276406bbc2e6df08beba708, State: Initialized, Role: FOLLOWER
Aug 10 19:02:41 dist-test-slave-znh6 krb5kdc[24389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388561, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:41.869395 24511 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c34f294276406bbc2e6df08beba708" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35609 } }
I20260810 19:02:41.869576 24511 raft_consensus.cc:388] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:41.869657 24511 raft_consensus.cc:482] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:41.869769 24511 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:41.870934 24511 raft_consensus.cc:504] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c34f294276406bbc2e6df08beba708" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35609 } }
I20260810 19:02:41.871302 24511 leader_election.cc:302] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 49c34f294276406bbc2e6df08beba708; no voters:
I20260810 19:02:41.871641 24511 leader_election.cc:288] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:41.871748 24523 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:41.871968 24523 raft_consensus.cc:686] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [term 1 LEADER]: Becoming Leader. State: Replica: 49c34f294276406bbc2e6df08beba708, State: Running, Role: LEADER
I20260810 19:02:41.872375 24523 consensus_queue.cc:237] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c34f294276406bbc2e6df08beba708" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35609 } }
I20260810 19:02:41.873059 24511 sys_catalog.cc:564] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:41.874287 24525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 49c34f294276406bbc2e6df08beba708. Latest consensus state: current_term: 1 leader_uuid: "49c34f294276406bbc2e6df08beba708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c34f294276406bbc2e6df08beba708" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35609 } } }
I20260810 19:02:41.874420 24525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:41.874740 24524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "49c34f294276406bbc2e6df08beba708" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "49c34f294276406bbc2e6df08beba708" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 35609 } } }
I20260810 19:02:41.874872 24542 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:41.874924 24524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:41.876262 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:41.861120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57496 (local address 127.27.147.190:35609)
0810 19:02:41.861627 (+ 507us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:41.861643 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:41.861678 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:02:41.862347 (+ 669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:41.862367 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:41.862672 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:41.863019 (+ 347us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:41.863980 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:41.865004 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:41.866213 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:41.866715 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:41.866925 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:41.869985 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:41.869991 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:41.870004 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:41.870038 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:41.872153 (+ 2115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:41.872683 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:41.872690 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:41.872696 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:41.872848 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:41.873211 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:41.873215 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:41.873216 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:41.873736 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:41.873957 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:41.874965 (+ 1008us) server_negotiation.cc:299] Negotiation successful
0810 19:02:41.875293 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":169,"threads_started":1}
I20260810 19:02:41.878129 24542 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:41.883737 24542 catalog_manager.cc:1347] Generated new cluster ID: d57808f0e6de4601868134cd249f2cdb
I20260810 19:02:41.883812 24542 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:41.894119 24542 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:41.895103 24542 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:41.901752 24542 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708: Generated new TSK 0
I20260810 19:02:41.902495 24542 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 19:02:41.938102 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35609
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:42.051887 24564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:42.052122 24564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:42.052163 24564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:42.056114 24564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:42.056162 24564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:42.056216 24564 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:02:42.061173 24564 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24564
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:42.062127 24564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:42.063098 24564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:42.072716 24572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:42.072728 24578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:42.072932 24564 server_base.cc:1034] running on GCE node
W20260810 19:02:42.072736 24575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:42.073377 24564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:42.074069 24564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:42.075266 24564 hybrid_clock.cc:648] HybridClock initialized: now 1786388562075246 us; error 31 us; skew 500 ppm
Aug 10 19:02:42 dist-test-slave-znh6 krb5kdc[24389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:42.077986 24564 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:02:42.078882 24564 webserver.cc:460] Webserver started at http://127.27.147.129:34447/ using document root <none> and password file <none>
I20260810 19:02:42.079437 24564 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:42.079535 24564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:42.079771 24564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:42.081746 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "305f24b6f02945cb872f885e4521230e"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "57e595ac4d9ea523baef28885a1bab1b"
server_key_iv: "b831c862d32ecf490595311205770ca8"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.082306 24564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "305f24b6f02945cb872f885e4521230e"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "57e595ac4d9ea523baef28885a1bab1b"
server_key_iv: "b831c862d32ecf490595311205770ca8"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.085932 24564 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 19:02:42.088287 24585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.089403 24564 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:02:42.089547 24564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "305f24b6f02945cb872f885e4521230e"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "57e595ac4d9ea523baef28885a1bab1b"
server_key_iv: "b831c862d32ecf490595311205770ca8"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.089656 24564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:42.117713 24564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:42.120951 24564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:42.121163 24564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:42.121795 24564 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:42.122659 24564 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:42.122736 24564 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.122839 24564 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:42.122889 24564 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.132509 24564 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:35141
I20260810 19:02:42.132546 24764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:35141 every 8 connection(s)
I20260810 19:02:42.133517 24564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:02:42.134565 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 24564
I20260810 19:02:42.134706 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:02:42.135082 28238 external_mini_cluster.cc:1428] Setting key 7dcfbf8667b48f0990c502a270318131
Aug 10 19:02:42 dist-test-slave-znh6 krb5kdc[24389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 19:02:42.149144 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.135532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:60977 (local address 127.27.147.190:35609)
0810 19:02:42.135707 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.135713 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.136621 (+ 908us) server_negotiation.cc:407] Connection header received
0810 19:02:42.137993 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.137997 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.138049 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.138129 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:42.139716 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.140245 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.142107 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.142341 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.142436 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.145350 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:42.145354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:42.145356 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:42.145385 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:42.146892 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.147565 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.147569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.147571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.147654 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.147992 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.147997 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.147998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.148174 (+ 176us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:42.148282 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.148818 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.148967 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:02:42.150138 24767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.135988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35609 (local address 127.27.147.129:60977)
0810 19:02:42.136476 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.136528 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.137783 (+ 1255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.138300 (+ 517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.138306 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.138799 (+ 493us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:42.139533 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.139556 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.140393 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.140399 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.141959 (+ 1560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.141974 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.142513 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.142521 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.142779 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.143443 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:42.143472 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:42.145186 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:42.147004 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:42.147010 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:42.147024 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:42.147459 (+ 435us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:42.147746 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:42.147750 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:42.147752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:42.147895 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:42.148317 (+ 422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:42.148330 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:42.148588 (+ 258us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.148775 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.149099 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269,"thread_start_us":136,"threads_started":1}
I20260810 19:02:42.151492 24765 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35609
I20260810 19:02:42.151726 24765 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:42.152350 24765 heartbeater.cc:511] Master 127.27.147.190:35609 requested a full tablet report, sending...
I20260810 19:02:42.154048 24460 ts_manager.cc:194] Registered new tserver with Master: 305f24b6f02945cb872f885e4521230e (127.27.147.129:35141)
I20260810 19:02:42.155246 24460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:60977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 19:02:42.166345 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35609
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:42.277345 24772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:42.277613 24772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:42.277706 24772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:42.281641 24772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:42.281718 24772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:42.281800 24772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:02:42.286919 24772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24772
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:42.287870 24772 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:42.288842 24772 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:42.298287 24784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:42.298347 24779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:42.298378 24780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:42.298517 24772 server_base.cc:1034] running on GCE node
I20260810 19:02:42.299067 24772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:42.299618 24772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:42.300781 24772 hybrid_clock.cc:648] HybridClock initialized: now 1786388562300764 us; error 29 us; skew 500 ppm
Aug 10 19:02:42 dist-test-slave-znh6 krb5kdc[24389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:42.303586 24772 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:02:42.304427 24772 webserver.cc:460] Webserver started at http://127.27.147.130:41813/ using document root <none> and password file <none>
I20260810 19:02:42.304968 24772 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:42.305028 24772 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:42.305259 24772 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:42.307245 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "464e43c368a14f47a8eb298946a58e88"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "c4add2547934c96a158dbd893a0a6f57"
server_key_iv: "e9dab56f024985ae8053cf398f1289a0"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.307796 24772 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "464e43c368a14f47a8eb298946a58e88"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "c4add2547934c96a158dbd893a0a6f57"
server_key_iv: "e9dab56f024985ae8053cf398f1289a0"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.312448 24772 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:02:42.314677 24794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.315857 24772 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:42.315976 24772 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "464e43c368a14f47a8eb298946a58e88"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "c4add2547934c96a158dbd893a0a6f57"
server_key_iv: "e9dab56f024985ae8053cf398f1289a0"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.316080 24772 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:42.327800 24772 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:42.330896 24772 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:42.331085 24772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:42.331679 24772 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:42.332458 24772 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:42.332504 24772 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.332567 24772 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:42.332605 24772 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.341332 24772 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:39603
I20260810 19:02:42.341394 24963 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:39603 every 8 connection(s)
I20260810 19:02:42.342145 24772 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
Aug 10 19:02:42 dist-test-slave-znh6 krb5kdc[24389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:42.352423 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 24772
I20260810 19:02:42.352586 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:02:42.352993 28238 external_mini_cluster.cc:1428] Setting key ee87f87e531ee3403fa797a31020457d
I20260810 19:02:42.355383 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.343860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:36883 (local address 127.27.147.190:35609)
0810 19:02:42.343992 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.343996 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.344760 (+ 764us) server_negotiation.cc:407] Connection header received
0810 19:02:42.346004 (+ 1244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.346007 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.346055 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.346135 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:42.347270 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.347813 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.348896 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.349120 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.349209 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.351563 (+ 2354us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:42.351566 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:42.351569 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:42.351592 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:42.353154 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.353703 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.353706 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.353707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.353815 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.354146 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.354149 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.354151 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.354397 (+ 246us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:42.354507 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.355138 (+ 631us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.355240 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 19:02:42.356451 24971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.344203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35609 (local address 127.27.147.130:36883)
0810 19:02:42.344624 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.344664 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.345795 (+ 1131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.346278 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.346283 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.346607 (+ 324us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:42.347123 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.347137 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.347924 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.347927 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.348765 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.348775 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.349255 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.349257 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.349388 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.349941 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:42.349958 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:42.351396 (+ 1438us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:42.353288 (+ 1892us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:42.353293 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:42.353303 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:42.353601 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:42.353915 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:42.353918 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:42.353919 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:42.354050 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:42.354531 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:42.354538 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:42.354813 (+ 275us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.355063 (+ 250us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.355428 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":137,"threads_started":1}
I20260810 19:02:42.357776 24964 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35609
I20260810 19:02:42.358021 24964 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:42.358489 24964 heartbeater.cc:511] Master 127.27.147.190:35609 requested a full tablet report, sending...
I20260810 19:02:42.359577 24460 ts_manager.cc:194] Registered new tserver with Master: 464e43c368a14f47a8eb298946a58e88 (127.27.147.130:39603)
I20260810 19:02:42.360472 24460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:36883
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 19:02:42.383148 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35609
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34635
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:42.498164 24975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:42.498389 24975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:42.498461 24975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:42.502404 24975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:42.502461 24975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:42.502548 24975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:42.507736 24975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34635
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:35609
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:42.508716 24975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:42.509670 24975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:42.518999 24984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:42.519013 24989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:42.519016 24982 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:42.519402 24975 server_base.cc:1034] running on GCE node
I20260810 19:02:42.519805 24975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:42.520502 24975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:42.521692 24975 hybrid_clock.cc:648] HybridClock initialized: now 1786388562521679 us; error 23 us; skew 500 ppm
Aug 10 19:02:42 dist-test-slave-znh6 krb5kdc[24389](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:42.524381 24975 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:02:42.525238 24975 webserver.cc:460] Webserver started at http://127.27.147.131:37853/ using document root <none> and password file <none>
I20260810 19:02:42.525780 24975 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:42.525870 24975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:42.526103 24975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:42.528100 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "af533c7283e84195ba897426ecf6c92b"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "1b0449641792fcc50b7d6c136e534184"
server_key_iv: "70e3d86e8502b55de9520bfec1eec966"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.528664 24975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "af533c7283e84195ba897426ecf6c92b"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "1b0449641792fcc50b7d6c136e534184"
server_key_iv: "70e3d86e8502b55de9520bfec1eec966"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.532171 24975 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:02:42.534660 24997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.535833 24975 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:42.535969 24975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "af533c7283e84195ba897426ecf6c92b"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "1b0449641792fcc50b7d6c136e534184"
server_key_iv: "70e3d86e8502b55de9520bfec1eec966"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.536073 24975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:42.554713 24975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:42.557343 24975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:42.557539 24975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:42.558079 24975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:42.558920 24975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:42.558987 24975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.559051 24975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:42.559103 24975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.568231 24975 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:32919
I20260810 19:02:42.568276 25176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:32919 every 8 connection(s)
I20260810 19:02:42.569064 24975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
Aug 10 19:02:42 dist-test-slave-znh6 krb5kdc[24389](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388562, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:42.579188 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 24975
I20260810 19:02:42.579335 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:02:42.579689 28238 external_mini_cluster.cc:1428] Setting key 312e634e3db8d6ef2157463944796bae
I20260810 19:02:42.582772 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.570736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:51847 (local address 127.27.147.190:35609)
0810 19:02:42.570922 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.570928 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.571718 (+ 790us) server_negotiation.cc:407] Connection header received
0810 19:02:42.573029 (+ 1311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.573032 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.573082 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.573157 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:42.574382 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.575033 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.576189 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.576418 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.576498 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.578947 (+ 2449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:42.578950 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:42.578952 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:42.578977 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:42.580606 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.581183 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.581189 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.581191 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.581282 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.581625 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.581630 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.581632 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.581884 (+ 252us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:42.581979 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.582485 (+ 506us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.582604 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:02:42.583519 25180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.571125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:35609 (local address 127.27.147.131:51847)
0810 19:02:42.571571 (+ 446us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.571612 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.572832 (+ 1220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.573312 (+ 480us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.573319 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.573642 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:42.574251 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.574266 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.575186 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.575189 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.576084 (+ 895us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.576093 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.576548 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.576551 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.576690 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.577272 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:42.577288 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:42.578696 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:42.580722 (+ 2026us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:42.580727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:42.580747 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:42.581079 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:42.581384 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:42.581387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:42.581390 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:42.581525 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:42.581994 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:42.582002 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:42.582245 (+ 243us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.582437 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.582733 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"thread_start_us":120,"threads_started":1}
I20260810 19:02:42.584529 25177 heartbeater.cc:347] Connected to a master server at 127.27.147.190:35609
I20260810 19:02:42.584770 25177 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:42.585232 25177 heartbeater.cc:511] Master 127.27.147.190:35609 requested a full tablet report, sending...
I20260810 19:02:42.586233 24460 ts_manager.cc:194] Registered new tserver with Master: af533c7283e84195ba897426ecf6c92b (127.27.147.131:32919)
I20260810 19:02:42.586793 24460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:51847
I20260810 19:02:42.593531 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:02:42.602698 24520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.595621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57508 (local address 127.27.147.190:35609)
0810 19:02:42.595734 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.595737 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.595899 (+ 162us) server_negotiation.cc:407] Connection header received
0810 19:02:42.595985 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.595987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.596021 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.596079 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:42.596712 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.597205 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.597970 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.598171 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.598239 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.599743 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:42.599747 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:42.599750 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:42.599782 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:42.600913 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.601424 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.601427 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.601431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.601506 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:42.601840 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:42.601843 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:42.601845 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:42.602032 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:42.602193 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.602466 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.602555 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:02:42.605458 24460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57508:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:02:42.608458 24460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:02:42.623123 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.617484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36112 (local address 127.27.147.130:39603)
0810 19:02:42.617980 (+ 496us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.617993 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.618359 (+ 366us) server_negotiation.cc:407] Connection header received
0810 19:02:42.618441 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.618453 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.618578 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.618727 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:42.618940 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.619932 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.621859 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.622427 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.622561 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.622690 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.622878 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.622982 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":131,"threads_started":1}
I20260810 19:02:42.623137 25196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.617087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:39603 (local address 127.0.0.1:36112)
0810 19:02:42.618112 (+ 1025us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.618123 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.618237 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.618706 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.618709 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.618727 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:42.618851 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.618859 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.620048 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.620054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.621736 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.621752 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.622554 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.622558 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.622663 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.622702 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.622844 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.623012 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":966,"mutex_wait_us":133,"thread_start_us":83,"threads_started":1}
I20260810 19:02:42.623163 25199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.616805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:35141 (local address 127.0.0.1:53672)
0810 19:02:42.617820 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.617835 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.617910 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.618843 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.618846 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.618868 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:42.618946 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.618953 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.620524 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.620531 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.621515 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.621525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.622665 (+ 1140us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.622669 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.622767 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.622791 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.622869 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.623006 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":947,"mutex_wait_us":3,"thread_start_us":106,"threads_started":1}
I20260810 19:02:42.623476 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.616827 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51590 (local address 127.27.147.131:32919)
0810 19:02:42.617243 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.617261 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.617593 (+ 332us) server_negotiation.cc:407] Connection header received
0810 19:02:42.617896 (+ 303us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.617902 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.618121 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.618250 (+ 129us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:42.618741 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.620016 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.621703 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.622521 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.622655 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.622778 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.623207 (+ 429us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.623335 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":331,"thread_start_us":145,"threads_started":1}
I20260810 19:02:42.623641 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.616717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:32919 (local address 127.0.0.1:51590)
0810 19:02:42.617457 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.617507 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.617699 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.618445 (+ 746us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.618451 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.618474 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:42.618598 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.618604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.620125 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.620128 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.621593 (+ 1465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.621604 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.622867 (+ 1263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.622870 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.622964 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.622984 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.623054 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.623114 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"mutex_wait_us":70,"thread_start_us":70,"threads_started":1}
I20260810 19:02:42.623967 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.617484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.27.147.129:35141)
0810 19:02:42.618314 (+ 830us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.618329 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.618354 (+ 25us) server_negotiation.cc:407] Connection header received
0810 19:02:42.618449 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.618454 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.618645 (+ 191us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.619106 (+ 461us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:42.619245 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.620349 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.621641 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.622521 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.623368 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.623501 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.623556 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.623800 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":74,"threads_started":1}
I20260810 19:02:42.626312 24655 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3f21581d3a48e1a0ff6bca1114f263 (DEFAULT_TABLE table=test-table [id=af2f415bf508476989cf59791bc061c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:42.626449 25075 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3f21581d3a48e1a0ff6bca1114f263 (DEFAULT_TABLE table=test-table [id=af2f415bf508476989cf59791bc061c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:42.626449 24863 tablet_service.cc:1467] Processing CreateTablet for tablet 6b3f21581d3a48e1a0ff6bca1114f263 (DEFAULT_TABLE table=test-table [id=af2f415bf508476989cf59791bc061c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:42.627377 24655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3f21581d3a48e1a0ff6bca1114f263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:42.627509 24863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3f21581d3a48e1a0ff6bca1114f263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:42.627509 25075 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6b3f21581d3a48e1a0ff6bca1114f263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:42.633807 25204 tablet_bootstrap.cc:492] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e: Bootstrap starting.
I20260810 19:02:42.635298 25206 tablet_bootstrap.cc:492] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Bootstrap starting.
I20260810 19:02:42.635468 25205 tablet_bootstrap.cc:492] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Bootstrap starting.
I20260810 19:02:42.636209 25204 tablet_bootstrap.cc:654] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:42.637305 25204 log.cc:824] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:42.637533 25205 tablet_bootstrap.cc:654] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:42.637533 25206 tablet_bootstrap.cc:654] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:42.638873 25205 log.cc:824] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:42.638870 25206 log.cc:824] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:42.639708 25204 tablet_bootstrap.cc:492] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e: No bootstrap required, opened a new log
I20260810 19:02:42.639923 25204 ts_tablet_manager.cc:1397] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.005s
I20260810 19:02:42.640969 25205 tablet_bootstrap.cc:492] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: No bootstrap required, opened a new log
I20260810 19:02:42.641144 25205 ts_tablet_manager.cc:1397] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:02:42.641160 25206 tablet_bootstrap.cc:492] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: No bootstrap required, opened a new log
I20260810 19:02:42.641321 25206 ts_tablet_manager.cc:1397] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 19:02:42.643086 25204 raft_consensus.cc:348] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.643334 25204 raft_consensus.cc:374] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:42.643410 25204 raft_consensus.cc:729] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 305f24b6f02945cb872f885e4521230e, State: Initialized, Role: FOLLOWER
I20260810 19:02:42.643864 25204 consensus_queue.cc:260] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.644531 24765 heartbeater.cc:503] Master 127.27.147.190:35609 was elected leader, sending a full tablet report...
I20260810 19:02:42.644865 25204 ts_tablet_manager.cc:1428] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260810 19:02:42.645036 25205 raft_consensus.cc:348] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.645036 25206 raft_consensus.cc:348] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.645284 25206 raft_consensus.cc:374] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:42.645284 25205 raft_consensus.cc:374] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:42.645344 25205 raft_consensus.cc:729] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 464e43c368a14f47a8eb298946a58e88, State: Initialized, Role: FOLLOWER
I20260810 19:02:42.645382 25206 raft_consensus.cc:729] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af533c7283e84195ba897426ecf6c92b, State: Initialized, Role: FOLLOWER
I20260810 19:02:42.645838 25206 consensus_queue.cc:260] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.645838 25205 consensus_queue.cc:260] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.646763 25205 ts_tablet_manager.cc:1428] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:02:42.646763 25206 ts_tablet_manager.cc:1428] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 19:02:42.647117 24964 heartbeater.cc:503] Master 127.27.147.190:35609 was elected leader, sending a full tablet report...
I20260810 19:02:42.647146 25177 heartbeater.cc:503] Master 127.27.147.190:35609 was elected leader, sending a full tablet report...
I20260810 19:02:42.673085 25212 raft_consensus.cc:482] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:42.673229 25212 raft_consensus.cc:504] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.674389 25212 leader_election.cc:288] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af533c7283e84195ba897426ecf6c92b (127.27.147.131:32919), 464e43c368a14f47a8eb298946a58e88 (127.27.147.130:39603)
I20260810 19:02:42.679356 25198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.674685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:47251 (local address 127.27.147.131:32919)
0810 19:02:42.674971 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.674977 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.674996 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:02:42.675057 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.675060 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.675125 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.675214 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:42.675573 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.676362 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.678080 (+ 1718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.678682 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.678807 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.678884 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.679165 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.679234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260810 19:02:42.679378 25203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.674834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:54727 (local address 127.27.147.130:39603)
0810 19:02:42.674978 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.674983 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.675117 (+ 134us) server_negotiation.cc:407] Connection header received
0810 19:02:42.675191 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.675197 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.675244 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.675318 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:42.675604 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.676379 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.678080 (+ 1701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.678685 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.678807 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.678884 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.679165 (+ 281us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.679234 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 19:02:42.679414 25217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.674635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:39603 (local address 127.27.147.129:54727)
0810 19:02:42.675022 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.675037 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.675122 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.675355 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.675357 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.675374 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:42.675498 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.675507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.676627 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.676631 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.677945 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.677957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.678838 (+ 881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.678842 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.678997 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.679052 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.679133 (+ 81us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.679244 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":309,"mutex_wait_us":71,"thread_start_us":119,"threads_started":1}
I20260810 19:02:42.679416 25216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.674556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:32919 (local address 127.27.147.129:47251)
0810 19:02:42.674799 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.674839 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:42.674950 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:42.675251 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:42.675253 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:42.675268 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:42.675422 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.675429 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.676505 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.676508 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.677945 (+ 1437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:42.677957 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.678831 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.678836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:42.679026 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.679066 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 19:02:42.679133 (+ 67us) client_negotiation.cc:240] Negotiation successful
0810 19:02:42.679249 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"mutex_wait_us":205,"thread_start_us":63,"threads_started":1}
I20260810 19:02:42.680054 24894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3f21581d3a48e1a0ff6bca1114f263" candidate_uuid: "305f24b6f02945cb872f885e4521230e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464e43c368a14f47a8eb298946a58e88" is_pre_election: true
I20260810 19:02:42.680095 25108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3f21581d3a48e1a0ff6bca1114f263" candidate_uuid: "305f24b6f02945cb872f885e4521230e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af533c7283e84195ba897426ecf6c92b" is_pre_election: true
I20260810 19:02:42.680317 24894 raft_consensus.cc:2454] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305f24b6f02945cb872f885e4521230e in term 0.
I20260810 19:02:42.680369 25108 raft_consensus.cc:2454] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 305f24b6f02945cb872f885e4521230e in term 0.
I20260810 19:02:42.680902 24589 leader_election.cc:302] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 305f24b6f02945cb872f885e4521230e, 464e43c368a14f47a8eb298946a58e88, af533c7283e84195ba897426ecf6c92b; no voters:
I20260810 19:02:42.681164 25212 raft_consensus.cc:2781] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:02:42.681265 25212 raft_consensus.cc:482] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:02:42.681344 25212 raft_consensus.cc:3037] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:42.682358 25212 raft_consensus.cc:504] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.682677 25212 leader_election.cc:288] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [CANDIDATE]: Term 1 election: Requested vote from peers af533c7283e84195ba897426ecf6c92b (127.27.147.131:32919), 464e43c368a14f47a8eb298946a58e88 (127.27.147.130:39603)
I20260810 19:02:42.683054 24894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3f21581d3a48e1a0ff6bca1114f263" candidate_uuid: "305f24b6f02945cb872f885e4521230e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "464e43c368a14f47a8eb298946a58e88"
I20260810 19:02:42.683048 25108 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6b3f21581d3a48e1a0ff6bca1114f263" candidate_uuid: "305f24b6f02945cb872f885e4521230e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af533c7283e84195ba897426ecf6c92b"
I20260810 19:02:42.683166 25108 raft_consensus.cc:3037] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:42.683166 24894 raft_consensus.cc:3037] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:42.684454 25108 raft_consensus.cc:2454] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305f24b6f02945cb872f885e4521230e in term 1.
I20260810 19:02:42.684454 24894 raft_consensus.cc:2454] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 305f24b6f02945cb872f885e4521230e in term 1.
I20260810 19:02:42.684751 24589 leader_election.cc:302] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 305f24b6f02945cb872f885e4521230e, 464e43c368a14f47a8eb298946a58e88, af533c7283e84195ba897426ecf6c92b; no voters:
I20260810 19:02:42.684955 25212 raft_consensus.cc:2781] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:42.685134 25212 raft_consensus.cc:686] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 1 LEADER]: Becoming Leader. State: Replica: 305f24b6f02945cb872f885e4521230e, State: Running, Role: LEADER
I20260810 19:02:42.685432 25212 consensus_queue.cc:237] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } }
I20260810 19:02:42.688246 24458 catalog_manager.cc:5495] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e reported cstate change: term changed from 0 to 1, leader changed from <none> to 305f24b6f02945cb872f885e4521230e (127.27.147.129). New cstate: current_term: 1 leader_uuid: "305f24b6f02945cb872f885e4521230e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "305f24b6f02945cb872f885e4521230e" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 35141 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 } health_report { overall_health: UNKNOWN } } }
I20260810 19:02:42.700243 25201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.697080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53674 (local address 127.27.147.129:35141)
0810 19:02:42.697199 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.697203 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.697235 (+ 32us) server_negotiation.cc:407] Connection header received
0810 19:02:42.697371 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.697373 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.697421 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.697502 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:42.697704 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.698161 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.698988 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.699187 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.699263 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.699507 (+ 244us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:42.699846 (+ 339us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:02:42.699960 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.700056 (+ 96us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.700085 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:02:42.705433 24894 raft_consensus.cc:1261] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 1 FOLLOWER]: Refusing update from remote peer 305f24b6f02945cb872f885e4521230e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:42.705441 25108 raft_consensus.cc:1261] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 1 FOLLOWER]: Refusing update from remote peer 305f24b6f02945cb872f885e4521230e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:42.706054 25212 consensus_queue.cc:1035] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [LEADER]: Connected to new peer: Peer: permanent_uuid: "af533c7283e84195ba897426ecf6c92b" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:42.706261 25218 consensus_queue.cc:1035] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [LEADER]: Connected to new peer: Peer: permanent_uuid: "464e43c368a14f47a8eb298946a58e88" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 39603 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:42.713425 25223 mvcc.cc:205] Tried to move back new op lower bound from 7317047552835334144 to 7317047552762003456. Current Snapshot: MvccSnapshot[applied={T|T < 7317047552835334144}]
I20260810 19:02:42.721524 24457 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:42.721685 24457 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57508:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:42.724087 24457 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 49c34f294276406bbc2e6df08beba708: Sending DeleteTablet for 3 replicas of tablet 6b3f21581d3a48e1a0ff6bca1114f263
I20260810 19:02:42.724793 24655 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3f21581d3a48e1a0ff6bca1114f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:42 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:53672
I20260810 19:02:42.724794 25075 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3f21581d3a48e1a0ff6bca1114f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:42 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:51590
I20260810 19:02:42.725267 25241 tablet_replica.cc:330] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e: stopping tablet replica
I20260810 19:02:42.725243 24863 tablet_service.cc:1514] Processing DeleteTablet for tablet 6b3f21581d3a48e1a0ff6bca1114f263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:42 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:36112
I20260810 19:02:42.725561 25241 raft_consensus.cc:2229] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:02:42.725656 25244 tablet_replica.cc:330] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: stopping tablet replica
I20260810 19:02:42.725914 25241 raft_consensus.cc:2258] T 6b3f21581d3a48e1a0ff6bca1114f263 P 305f24b6f02945cb872f885e4521230e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:42.725940 25244 raft_consensus.cc:2229] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:42.726199 25244 raft_consensus.cc:2258] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:42.726248 25239 tablet_replica.cc:330] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: stopping tablet replica
I20260810 19:02:42.726516 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 24564
I20260810 19:02:42.726540 25239 raft_consensus.cc:2229] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:42.728230 25244 ts_tablet_manager.cc:1905] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:42.731873 25244 ts_tablet_manager.cc:1918] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:42.732005 25244 log.cc:1196] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/6b3f21581d3a48e1a0ff6bca1114f263
I20260810 19:02:42.732314 25239 raft_consensus.cc:2258] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:42.732373 25244 ts_tablet_manager.cc:1939] T 6b3f21581d3a48e1a0ff6bca1114f263 P 464e43c368a14f47a8eb298946a58e88: Deleting consensus metadata
I20260810 19:02:42.733480 24438 catalog_manager.cc:4841] TS 464e43c368a14f47a8eb298946a58e88 (127.27.147.130:39603): tablet 6b3f21581d3a48e1a0ff6bca1114f263 (table test-table [id=af2f415bf508476989cf59791bc061c8]) successfully deleted
I20260810 19:02:42.735806 25239 ts_tablet_manager.cc:1905] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:42.738339 25239 ts_tablet_manager.cc:1918] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:42.738445 25239 log.cc:1196] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/6b3f21581d3a48e1a0ff6bca1114f263
I20260810 19:02:42.738757 25239 ts_tablet_manager.cc:1939] T 6b3f21581d3a48e1a0ff6bca1114f263 P af533c7283e84195ba897426ecf6c92b: Deleting consensus metadata
I20260810 19:02:42.739887 24436 catalog_manager.cc:4841] TS af533c7283e84195ba897426ecf6c92b (127.27.147.131:32919): tablet 6b3f21581d3a48e1a0ff6bca1114f263 (table test-table [id=af2f415bf508476989cf59791bc061c8]) successfully deleted
W20260810 19:02:42.742465 24435 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:35141 (error 108)
I20260810 19:02:42.742612 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 24772
I20260810 19:02:42.746160 25195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.744790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:35141 (local address 127.0.0.1:53684)
0810 19:02:42.745992 (+ 1202us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:42.746069 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:35141: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1095}
W20260810 19:02:42.747382 24435 catalog_manager.cc:4584] TS 305f24b6f02945cb872f885e4521230e (127.27.147.129:35141): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6b3f21581d3a48e1a0ff6bca1114f263: Network error: Client connection negotiation failed: client connection to 127.27.147.129:35141: connect: Connection refused (error 111)
I20260810 19:02:42.752604 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 24975
I20260810 19:02:42.763815 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 24410
2026-08-10T19:02:42Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2488 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T19:02:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:42Z Disabled control of system clock
I20260810 19:02:42.798518 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34131
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:34131
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:02:42.910884 25254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:42.911114 25254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:42.911149 25254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:42.915129 25254 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:42.915174 25254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:42.915190 25254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:42.915206 25254 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:42.915248 25254 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:42.919875 25254 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:34131
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:42.920851 25254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:42.922138 25254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:42.930720 25261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:42.930727 25262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:42.930830 25254 server_base.cc:1034] running on GCE node
W20260810 19:02:42.930732 25265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:42.931528 25254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:42.932565 25254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:42.933741 25254 hybrid_clock.cc:648] HybridClock initialized: now 1786388562933722 us; error 32 us; skew 500 ppm
I20260810 19:02:42.935593 25254 webserver.cc:460] Webserver started at http://127.27.147.190:44969/ using document root <none> and password file <none>
I20260810 19:02:42.936158 25254 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:42.936250 25254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:42.936497 25254 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:42.938444 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "095cd2e9c0cc469ab50c8b78d0a1006d"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "89b9c6ec7eb8a4a943e86c82dfd7ee6a"
server_key_iv: "9e3a4449812603f68824cd7c19671842"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.939100 25254 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "095cd2e9c0cc469ab50c8b78d0a1006d"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "89b9c6ec7eb8a4a943e86c82dfd7ee6a"
server_key_iv: "9e3a4449812603f68824cd7c19671842"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.942628 25254 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:02:42.945032 25274 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:42.946244 25254 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:42.946388 25254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "095cd2e9c0cc469ab50c8b78d0a1006d"
format_stamp: "Formatted at 2026-08-10 19:02:42 on dist-test-slave-znh6"
server_key: "89b9c6ec7eb8a4a943e86c82dfd7ee6a"
server_key_iv: "9e3a4449812603f68824cd7c19671842"
server_key_version: "encryptionkey@0"
I20260810 19:02:42.946511 25254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:42.965721 25254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:42.966351 25254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:42.966557 25254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:42.973738 25254 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:34131
I20260810 19:02:42.973744 25361 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:34131 every 8 connection(s)
I20260810 19:02:42.974658 25254 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:42.977932 25362 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:42.983525 25362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d: Bootstrap starting.
I20260810 19:02:42.984503 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 25254
I20260810 19:02:42.984635 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:42.984985 28238 external_mini_cluster.cc:1428] Setting key a393ecc654928e8369c246a8f5fdc440
I20260810 19:02:42.986584 25362 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:42.987381 25362 log.cc:824] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:42.989316 25362 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d: No bootstrap required, opened a new log
I20260810 19:02:42.992799 25362 raft_consensus.cc:348] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34131 } }
I20260810 19:02:42.993009 25362 raft_consensus.cc:374] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:42.993070 25362 raft_consensus.cc:729] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 095cd2e9c0cc469ab50c8b78d0a1006d, State: Initialized, Role: FOLLOWER
I20260810 19:02:42.993568 25362 consensus_queue.cc:260] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34131 } }
I20260810 19:02:42.993732 25362 raft_consensus.cc:388] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:42.993820 25362 raft_consensus.cc:482] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:42.993944 25362 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:42.994519 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:42.986169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37622 (local address 127.27.147.190:34131)
0810 19:02:42.986941 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:42.986957 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:42.987034 (+ 77us) server_negotiation.cc:407] Connection header received
0810 19:02:42.987794 (+ 760us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:42.987819 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:42.988239 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:42.988602 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:42.989027 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.990063 (+ 1036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.991087 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:42.991558 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:42.991737 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:42.992621 (+ 884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:42.992626 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:42.992638 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:42.993037 (+ 399us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:42.993111 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:42.993156 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:42.993267 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:42.993398 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 19:02:42.993666 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":321,"thread_start_us":151,"threads_started":1}
I20260810 19:02:42.994938 25362 raft_consensus.cc:504] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34131 } }
I20260810 19:02:42.995277 25362 leader_election.cc:302] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 095cd2e9c0cc469ab50c8b78d0a1006d; no voters:
I20260810 19:02:42.995622 25362 leader_election.cc:288] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:42.995731 25369 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:42.995949 25369 raft_consensus.cc:686] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [term 1 LEADER]: Becoming Leader. State: Replica: 095cd2e9c0cc469ab50c8b78d0a1006d, State: Running, Role: LEADER
I20260810 19:02:42.996274 25369 consensus_queue.cc:237] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34131 } }
I20260810 19:02:42.996827 25362 sys_catalog.cc:564] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:42.997573 25371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 095cd2e9c0cc469ab50c8b78d0a1006d. Latest consensus state: current_term: 1 leader_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34131 } } }
I20260810 19:02:42.997776 25371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:42.997957 25370 sys_catalog.cc:455] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "095cd2e9c0cc469ab50c8b78d0a1006d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34131 } } }
I20260810 19:02:42.998070 25370 sys_catalog.cc:458] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:42.998240 25378 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:43.001349 25378 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:43.006541 25378 catalog_manager.cc:1347] Generated new cluster ID: 0bd03a14afd04b8aae2337b7715870a0
I20260810 19:02:43.006613 25378 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:43.029007 25378 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:43.030123 25378 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:43.041553 25378 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d: Generated new TSK 0
I20260810 19:02:43.042277 25378 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:43.052984 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34131
--builtin_ntp_servers=127.27.147.148:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:02:43.167536 25391 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:43.167778 25391 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:43.167850 25391 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:43.171847 25391 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:43.171893 25391 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:43.171968 25391 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:02:43.176625 25391 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25391
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:43.177587 25391 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:43.178920 25391 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:43.188318 25400 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:43.188506 25391 server_base.cc:1034] running on GCE node
W20260810 19:02:43.188387 25404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:43.188649 25401 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:43.189095 25391 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:43.189725 25391 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:43.190907 25391 hybrid_clock.cc:648] HybridClock initialized: now 1786388563190866 us; error 50 us; skew 500 ppm
I20260810 19:02:43.192633 25391 webserver.cc:460] Webserver started at http://127.27.147.129:41897/ using document root <none> and password file <none>
I20260810 19:02:43.193177 25391 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:43.193269 25391 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:43.193509 25391 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:43.195550 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "2dc00b17497d4582a2387271af5444b2"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "9b772001aea24d56f6ecc8496fefd4c4"
server_key_iv: "718e5107f45c8201ad218fbcd613b4e9"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.196107 25391 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "2dc00b17497d4582a2387271af5444b2"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "9b772001aea24d56f6ecc8496fefd4c4"
server_key_iv: "718e5107f45c8201ad218fbcd613b4e9"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.200007 25391 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:02:43.202316 25414 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.203528 25391 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:43.203665 25391 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "2dc00b17497d4582a2387271af5444b2"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "9b772001aea24d56f6ecc8496fefd4c4"
server_key_iv: "718e5107f45c8201ad218fbcd613b4e9"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.203770 25391 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:43.221402 25391 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:43.222005 25391 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:43.222193 25391 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:43.222739 25391 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:43.223670 25391 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:43.223743 25391 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.223814 25391 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:43.223863 25391 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.232836 25391 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:37711
I20260810 19:02:43.232854 25577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:37711 every 8 connection(s)
I20260810 19:02:43.233662 25391 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:02:43.240104 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 25391
I20260810 19:02:43.240257 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:02:43.240547 28238 external_mini_cluster.cc:1428] Setting key b15d0a2b8488677cdcc6e26345c5feee
I20260810 19:02:43.242712 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34131
--builtin_ntp_servers=127.27.147.148:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:02:43.242980 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.235509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:40047 (local address 127.27.147.190:34131)
0810 19:02:43.235666 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.235670 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.236699 (+ 1029us) server_negotiation.cc:407] Connection header received
0810 19:02:43.238071 (+ 1372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.238074 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.238122 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.238198 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:43.239089 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.239646 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.240913 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.241169 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.241258 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.242389 (+ 1131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:43.242393 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:43.242397 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:43.242412 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:43.242440 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:43.242454 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:43.242541 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.242691 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.242765 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 19:02:43.243732 25581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.235908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34131 (local address 127.27.147.129:40047)
0810 19:02:43.236557 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.236596 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.237839 (+ 1243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.238321 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.238327 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.238404 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:43.238951 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.238966 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.239763 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.239766 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.240786 (+ 1020us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.240797 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.241335 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.241339 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.241492 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.242132 (+ 640us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:43.242134 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:43.242172 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:43.242232 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:43.242546 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:43.242554 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:43.242558 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.242652 (+ 94us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.242962 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":100,"threads_started":1}
I20260810 19:02:43.244961 25578 heartbeater.cc:347] Connected to a master server at 127.27.147.190:34131
I20260810 19:02:43.245214 25578 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:43.245828 25578 heartbeater.cc:511] Master 127.27.147.190:34131 requested a full tablet report, sending...
I20260810 19:02:43.247398 25297 ts_manager.cc:194] Registered new tserver with Master: 2dc00b17497d4582a2387271af5444b2 (127.27.147.129:37711)
I20260810 19:02:43.248558 25297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.129:40047
W20260810 19:02:43.358194 25582 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:43.358506 25582 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:43.358580 25582 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:43.362743 25582 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:43.362850 25582 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:43.362947 25582 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:02:43.367714 25582 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25582
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:43.368801 25582 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:43.370314 25582 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:43.380115 25590 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:43.380167 25591 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:43.380175 25593 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:43.380754 25582 server_base.cc:1034] running on GCE node
I20260810 19:02:43.381186 25582 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:43.381800 25582 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:43.382947 25582 hybrid_clock.cc:648] HybridClock initialized: now 1786388563382918 us; error 47 us; skew 500 ppm
I20260810 19:02:43.384831 25582 webserver.cc:460] Webserver started at http://127.27.147.130:33073/ using document root <none> and password file <none>
I20260810 19:02:43.385421 25582 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:43.385484 25582 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:43.385699 25582 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:43.387796 25582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "bffeb10610b74a5b8737831dfb781eb6"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "efdc32e20e3b0393773a072e9d158303"
server_key_iv: "7b5b037f6bcda1583d59179d4424fe3c"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.388500 25582 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "bffeb10610b74a5b8737831dfb781eb6"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "efdc32e20e3b0393773a072e9d158303"
server_key_iv: "7b5b037f6bcda1583d59179d4424fe3c"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.392343 25582 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 19:02:43.394724 25602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.395923 25582 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:02:43.396060 25582 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "bffeb10610b74a5b8737831dfb781eb6"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "efdc32e20e3b0393773a072e9d158303"
server_key_iv: "7b5b037f6bcda1583d59179d4424fe3c"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.396173 25582 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:43.417979 25582 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:43.418699 25582 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:43.418936 25582 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:43.419561 25582 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:43.420542 25582 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:43.420619 25582 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.420691 25582 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:43.420738 25582 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.429975 25582 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:34765
I20260810 19:02:43.430006 25772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:34765 every 8 connection(s)
I20260810 19:02:43.430960 25582 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:02:43.440207 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.432895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:33081 (local address 127.27.147.190:34131)
0810 19:02:43.433088 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.433093 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.434106 (+ 1013us) server_negotiation.cc:407] Connection header received
0810 19:02:43.435566 (+ 1460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.435569 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.435615 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.435688 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:43.436688 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.437220 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.438305 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.438516 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.438592 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.439771 (+ 1179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:43.439775 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:43.439776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:43.439789 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:43.439816 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:43.439825 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:43.439886 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.440036 (+ 150us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.440089 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 19:02:43.441114 25775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.433299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34131 (local address 127.27.147.130:33081)
0810 19:02:43.433958 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.433998 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.435370 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.435849 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.435856 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.435996 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:43.436556 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.436569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.437327 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.437329 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.438198 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.438207 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.438644 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.438646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.438784 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.439529 (+ 745us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:43.439530 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:43.439568 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:43.439627 (+ 59us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:43.439908 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:43.439915 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:43.439920 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.440015 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.440258 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":118,"threads_started":1}
I20260810 19:02:43.441219 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 25582
I20260810 19:02:43.441359 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:02:43.441747 28238 external_mini_cluster.cc:1428] Setting key c5f618c8241129b95d102d04b73fa929
I20260810 19:02:43.442405 25773 heartbeater.cc:347] Connected to a master server at 127.27.147.190:34131
I20260810 19:02:43.442653 25773 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:43.443308 25773 heartbeater.cc:511] Master 127.27.147.190:34131 requested a full tablet report, sending...
I20260810 19:02:43.444062 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34131
--builtin_ntp_servers=127.27.147.148:42051
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:02:43.444360 25297 ts_manager.cc:194] Registered new tserver with Master: bffeb10610b74a5b8737831dfb781eb6 (127.27.147.130:34765)
I20260810 19:02:43.444914 25297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.130:33081
W20260810 19:02:43.556149 25776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:43.556381 25776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:43.556419 25776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:43.560410 25776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:43.560453 25776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:43.560534 25776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:43.565119 25776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42051
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:34131
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:43.566069 25776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:43.567396 25776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:43.576815 25791 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:43.576825 25783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:43.576961 25776 server_base.cc:1034] running on GCE node
W20260810 19:02:43.576881 25785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:43.577509 25776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:43.578073 25776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:43.579257 25776 hybrid_clock.cc:648] HybridClock initialized: now 1786388563579225 us; error 40 us; skew 500 ppm
I20260810 19:02:43.581125 25776 webserver.cc:460] Webserver started at http://127.27.147.131:39027/ using document root <none> and password file <none>
I20260810 19:02:43.581737 25776 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:43.581801 25776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:43.581965 25776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:43.583935 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "2686ffe8075a4398b2454dedd8d68c00"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "e19278c14e5ccdb3d1563fa14eabaf50"
server_key_iv: "db231d0b980aa25987b882eb6909d737"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.584439 25776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "2686ffe8075a4398b2454dedd8d68c00"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "e19278c14e5ccdb3d1563fa14eabaf50"
server_key_iv: "db231d0b980aa25987b882eb6909d737"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.587834 25776 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:02:43.590037 25800 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.591274 25776 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:02:43.591365 25776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "2686ffe8075a4398b2454dedd8d68c00"
format_stamp: "Formatted at 2026-08-10 19:02:43 on dist-test-slave-znh6"
server_key: "e19278c14e5ccdb3d1563fa14eabaf50"
server_key_iv: "db231d0b980aa25987b882eb6909d737"
server_key_version: "encryptionkey@0"
I20260810 19:02:43.591452 25776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:43.621441 25776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:43.622066 25776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:43.622216 25776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:43.622769 25776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:43.623775 25776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:43.623824 25776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.623864 25776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:43.623879 25776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:43.632531 25776 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:39321
I20260810 19:02:43.632560 25967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:39321 every 8 connection(s)
I20260810 19:02:43.633440 25776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:02:43.641433 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 25776
I20260810 19:02:43.641551 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:02:43.641831 28238 external_mini_cluster.cc:1428] Setting key cbb852eb6476e799fb7c158b6481857a
I20260810 19:02:43.642704 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.635353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:45387 (local address 127.27.147.190:34131)
0810 19:02:43.635506 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.635510 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.636590 (+ 1080us) server_negotiation.cc:407] Connection header received
0810 19:02:43.638024 (+ 1434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.638027 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.638078 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.638153 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:43.639056 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.639582 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.640683 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.640899 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.640976 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.642175 (+ 1199us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:43.642179 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:43.642182 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:43.642195 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:43.642225 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:43.642233 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:43.642316 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.642495 (+ 179us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.642561 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:02:43.643683 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.635689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34131 (local address 127.27.147.131:45387)
0810 19:02:43.636396 (+ 707us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.636454 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.637804 (+ 1350us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.638325 (+ 521us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.638332 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.638414 (+ 82us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:43.638915 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.638938 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.639697 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.639699 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.640573 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.640582 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.641038 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.641041 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.641187 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.641858 (+ 671us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:43.641860 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:43.641925 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:43.642014 (+ 89us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:43.642319 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:43.642331 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:43.642341 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.642485 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.642756 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"thread_start_us":121,"threads_started":1}
I20260810 19:02:43.644752 25968 heartbeater.cc:347] Connected to a master server at 127.27.147.190:34131
I20260810 19:02:43.644996 25968 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:43.645503 25968 heartbeater.cc:511] Master 127.27.147.190:34131 requested a full tablet report, sending...
I20260810 19:02:43.646514 25297 ts_manager.cc:194] Registered new tserver with Master: 2686ffe8075a4398b2454dedd8d68c00 (127.27.147.131:39321)
I20260810 19:02:43.647150 25297 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.131:45387
I20260810 19:02:43.655783 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:02:43.661310 25365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.657723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37628 (local address 127.27.147.190:34131)
0810 19:02:43.657844 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.657848 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.658039 (+ 191us) server_negotiation.cc:407] Connection header received
0810 19:02:43.658135 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.658137 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.658174 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.658238 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:43.658442 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.659011 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.659820 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.660033 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.660113 (+ 80us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.660928 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:43.660931 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:43.660933 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:43.660943 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:43.660958 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:43.660965 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:43.661055 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.661183 (+ 128us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.661212 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:02:43.664249 25297 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37628:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:02:43.667240 25297 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:02:43.681077 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.675176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:34765 (local address 127.0.0.1:34444)
0810 19:02:43.676048 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.676095 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.676255 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.676983 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.676989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.677011 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:43.677137 (+ 126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.677149 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.678266 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.678268 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.679336 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.679347 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.680583 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.680586 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.680665 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.680691 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.680811 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.680933 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":88,"threads_started":1}
I20260810 19:02:43.681123 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.675313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34444 (local address 127.27.147.130:34765)
0810 19:02:43.675757 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.675769 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.676196 (+ 427us) server_negotiation.cc:407] Connection header received
0810 19:02:43.676416 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.676423 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.676643 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.676773 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:43.677279 (+ 506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.678151 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.679473 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.680276 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.680396 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.680529 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.680869 (+ 340us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.680963 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":116,"threads_started":1}
I20260810 19:02:43.681391 25984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.675583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37711 (local address 127.0.0.1:54982)
0810 19:02:43.676464 (+ 881us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.676477 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.676546 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.677345 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.677348 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.677363 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:43.677522 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.677530 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.678895 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.678898 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.679974 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.679983 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.680993 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.680997 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.681085 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.681113 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.681166 (+ 53us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.681254 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":815,"mutex_wait_us":51,"thread_start_us":94,"threads_started":1}
I20260810 19:02:43.681636 25986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.675332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:39321 (local address 127.0.0.1:60310)
0810 19:02:43.676096 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.676108 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.676235 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.677245 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.677251 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.677273 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:43.677418 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.677425 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.679061 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.679066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.680223 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.680237 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.681272 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.681276 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.681359 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.681382 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.681448 (+ 66us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.681515 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"mutex_wait_us":13,"thread_start_us":112,"threads_started":1}
I20260810 19:02:43.682247 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.676469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54982 (local address 127.27.147.129:37711)
0810 19:02:43.676933 (+ 464us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.676942 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.676962 (+ 20us) server_negotiation.cc:407] Connection header received
0810 19:02:43.677026 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.677030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.677164 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.677453 (+ 289us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:43.677628 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.678739 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.680097 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.680828 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.681588 (+ 760us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.681928 (+ 340us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.681981 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.682078 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":141,"threads_started":1}
I20260810 19:02:43.682411 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.676016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60310 (local address 127.27.147.131:39321)
0810 19:02:43.676800 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.676812 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.676833 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:02:43.676910 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.676918 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.677090 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.677249 (+ 159us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:43.677552 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.678739 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.680367 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.681145 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.681930 (+ 785us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.682088 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.682142 (+ 54us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.682222 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":680,"thread_start_us":134,"threads_started":1}
I20260810 19:02:43.684430 25676 tablet_service.cc:1467] Processing CreateTablet for tablet 4907e04fbbd74670885ef537f9c561a0 (DEFAULT_TABLE table=test-table [id=69101cf6997245bdab0aba479a6f33ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:43.684815 25867 tablet_service.cc:1467] Processing CreateTablet for tablet 4907e04fbbd74670885ef537f9c561a0 (DEFAULT_TABLE table=test-table [id=69101cf6997245bdab0aba479a6f33ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:43.685086 25481 tablet_service.cc:1467] Processing CreateTablet for tablet 4907e04fbbd74670885ef537f9c561a0 (DEFAULT_TABLE table=test-table [id=69101cf6997245bdab0aba479a6f33ae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:43.685586 25676 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4907e04fbbd74670885ef537f9c561a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:43.685802 25867 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4907e04fbbd74670885ef537f9c561a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:43.686043 25481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4907e04fbbd74670885ef537f9c561a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:43.692318 25996 tablet_bootstrap.cc:492] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Bootstrap starting.
I20260810 19:02:43.693234 25999 tablet_bootstrap.cc:492] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Bootstrap starting.
I20260810 19:02:43.694733 25996 tablet_bootstrap.cc:654] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:43.695111 25997 tablet_bootstrap.cc:492] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: Bootstrap starting.
I20260810 19:02:43.695444 25999 tablet_bootstrap.cc:654] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:43.695650 25996 log.cc:824] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:43.696244 25999 log.cc:824] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:43.697265 25997 tablet_bootstrap.cc:654] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:43.697942 25996 tablet_bootstrap.cc:492] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: No bootstrap required, opened a new log
I20260810 19:02:43.698143 25997 log.cc:824] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:43.698182 25996 ts_tablet_manager.cc:1397] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:02:43.698426 25999 tablet_bootstrap.cc:492] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: No bootstrap required, opened a new log
I20260810 19:02:43.698632 25999 ts_tablet_manager.cc:1397] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260810 19:02:43.700081 25997 tablet_bootstrap.cc:492] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: No bootstrap required, opened a new log
I20260810 19:02:43.700280 25997 ts_tablet_manager.cc:1397] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260810 19:02:43.702596 25996 raft_consensus.cc:348] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.702596 25999 raft_consensus.cc:348] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.702872 25996 raft_consensus.cc:374] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:43.702872 25999 raft_consensus.cc:374] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:43.702937 25999 raft_consensus.cc:729] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bffeb10610b74a5b8737831dfb781eb6, State: Initialized, Role: FOLLOWER
I20260810 19:02:43.702948 25996 raft_consensus.cc:729] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2686ffe8075a4398b2454dedd8d68c00, State: Initialized, Role: FOLLOWER
I20260810 19:02:43.703418 25996 consensus_queue.cc:260] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.703409 25999 consensus_queue.cc:260] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.703766 25997 raft_consensus.cc:348] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.704041 25997 raft_consensus.cc:374] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:43.704106 25997 raft_consensus.cc:729] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc00b17497d4582a2387271af5444b2, State: Initialized, Role: FOLLOWER
I20260810 19:02:43.704432 25996 ts_tablet_manager.cc:1428] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Time spent starting tablet: real 0.006s user 0.002s sys 0.004s
I20260810 19:02:43.704505 25773 heartbeater.cc:503] Master 127.27.147.190:34131 was elected leader, sending a full tablet report...
I20260810 19:02:43.704541 25997 consensus_queue.cc:260] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.705111 25999 ts_tablet_manager.cc:1428] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Time spent starting tablet: real 0.006s user 0.000s sys 0.006s
I20260810 19:02:43.705150 25578 heartbeater.cc:503] Master 127.27.147.190:34131 was elected leader, sending a full tablet report...
I20260810 19:02:43.705457 25997 ts_tablet_manager.cc:1428] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 19:02:43.705550 25968 heartbeater.cc:503] Master 127.27.147.190:34131 was elected leader, sending a full tablet report...
W20260810 19:02:43.734833 25580 tablet.cc:2369] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:02:43.828027 26004 raft_consensus.cc:482] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:43.828217 26004 raft_consensus.cc:504] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.829257 26004 leader_election.cc:288] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bffeb10610b74a5b8737831dfb781eb6 (127.27.147.130:34765), 2dc00b17497d4582a2387271af5444b2 (127.27.147.129:37711)
I20260810 19:02:43.833173 25985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.829599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:45189 (local address 127.27.147.130:34765)
0810 19:02:43.829709 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.829714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.829732 (+ 18us) server_negotiation.cc:407] Connection header received
0810 19:02:43.829803 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.829808 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.829863 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.829964 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:43.830346 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.830997 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.832121 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.832579 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.832671 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.832711 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.832961 (+ 250us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.833045 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:02:43.833169 25974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.829492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:34765 (local address 127.27.147.131:45189)
0810 19:02:43.829597 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.829614 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.829709 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.829986 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.829988 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.830027 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:43.830191 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.830198 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.831104 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.831108 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.831987 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.831997 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.832705 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.832709 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.832820 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.832847 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.832934 (+ 87us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.833008 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1,"spinlock_wait_cycles":7040}
I20260810 19:02:43.833302 25993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.829586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:38655 (local address 127.27.147.129:37711)
0810 19:02:43.829709 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.829714 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.830054 (+ 340us) server_negotiation.cc:407] Connection header received
0810 19:02:43.830192 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.830195 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.830259 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.830344 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:43.830640 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.831247 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.832283 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.832721 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.832835 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.832880 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.833106 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.833176 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260810 19:02:43.833516 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.829492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37711 (local address 127.27.147.131:38655)
0810 19:02:43.829945 (+ 453us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.829961 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:43.830074 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:43.830364 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:43.830367 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:43.830397 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:43.830521 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.830528 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.831361 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.831364 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.832155 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:43.832165 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.832852 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.832855 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:43.832964 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.832990 (+ 26us) client_negotiation.cc:769] Sending connection context
0810 19:02:43.833065 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 19:02:43.833130 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":117,"thread_start_us":115,"threads_started":1}
I20260810 19:02:43.833894 25702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4907e04fbbd74670885ef537f9c561a0" candidate_uuid: "2686ffe8075a4398b2454dedd8d68c00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffeb10610b74a5b8737831dfb781eb6" is_pre_election: true
I20260810 19:02:43.834117 25506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4907e04fbbd74670885ef537f9c561a0" candidate_uuid: "2686ffe8075a4398b2454dedd8d68c00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc00b17497d4582a2387271af5444b2" is_pre_election: true
I20260810 19:02:43.834218 25702 raft_consensus.cc:2454] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2686ffe8075a4398b2454dedd8d68c00 in term 0.
I20260810 19:02:43.834414 25506 raft_consensus.cc:2454] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2686ffe8075a4398b2454dedd8d68c00 in term 0.
I20260810 19:02:43.834911 25802 leader_election.cc:302] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 2686ffe8075a4398b2454dedd8d68c00, 2dc00b17497d4582a2387271af5444b2, bffeb10610b74a5b8737831dfb781eb6; no voters:
I20260810 19:02:43.835196 26004 raft_consensus.cc:2781] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:02:43.835266 26004 raft_consensus.cc:482] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:02:43.835295 26004 raft_consensus.cc:3037] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:43.836331 26004 raft_consensus.cc:504] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.836685 26004 leader_election.cc:288] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [CANDIDATE]: Term 1 election: Requested vote from peers bffeb10610b74a5b8737831dfb781eb6 (127.27.147.130:34765), 2dc00b17497d4582a2387271af5444b2 (127.27.147.129:37711)
I20260810 19:02:43.837067 25506 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4907e04fbbd74670885ef537f9c561a0" candidate_uuid: "2686ffe8075a4398b2454dedd8d68c00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2dc00b17497d4582a2387271af5444b2"
I20260810 19:02:43.837198 25506 raft_consensus.cc:3037] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:43.837235 25702 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4907e04fbbd74670885ef537f9c561a0" candidate_uuid: "2686ffe8075a4398b2454dedd8d68c00" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bffeb10610b74a5b8737831dfb781eb6"
I20260810 19:02:43.837342 25702 raft_consensus.cc:3037] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:43.838526 25702 raft_consensus.cc:2454] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2686ffe8075a4398b2454dedd8d68c00 in term 1.
I20260810 19:02:43.838526 25506 raft_consensus.cc:2454] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2686ffe8075a4398b2454dedd8d68c00 in term 1.
I20260810 19:02:43.838896 25804 leader_election.cc:302] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2686ffe8075a4398b2454dedd8d68c00, 2dc00b17497d4582a2387271af5444b2; no voters:
I20260810 19:02:43.839040 26004 raft_consensus.cc:2781] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:43.839244 26004 raft_consensus.cc:686] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 1 LEADER]: Becoming Leader. State: Replica: 2686ffe8075a4398b2454dedd8d68c00, State: Running, Role: LEADER
I20260810 19:02:43.839561 26004 consensus_queue.cc:237] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } }
I20260810 19:02:43.842476 25297 catalog_manager.cc:5495] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2686ffe8075a4398b2454dedd8d68c00 (127.27.147.131). New cstate: current_term: 1 leader_uuid: "2686ffe8075a4398b2454dedd8d68c00" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2686ffe8075a4398b2454dedd8d68c00" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 39321 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 } health_report { overall_health: UNKNOWN } } }
I20260810 19:02:43.868032 25991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.864836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60314 (local address 127.27.147.131:39321)
0810 19:02:43.864989 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:43.864992 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:43.865004 (+ 12us) server_negotiation.cc:407] Connection header received
0810 19:02:43.865060 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:43.865062 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:43.865110 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:43.865193 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:43.865415 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.865889 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.866679 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:43.866950 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:43.867028 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:43.867322 (+ 294us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:43.867706 (+ 384us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:02:43.867817 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:43.867903 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 19:02:43.867933 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 19:02:43.873605 25702 raft_consensus.cc:1261] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 1 FOLLOWER]: Refusing update from remote peer 2686ffe8075a4398b2454dedd8d68c00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:43.873623 25506 raft_consensus.cc:1261] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 1 FOLLOWER]: Refusing update from remote peer 2686ffe8075a4398b2454dedd8d68c00: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:43.874293 26004 consensus_queue.cc:1035] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2dc00b17497d4582a2387271af5444b2" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 37711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:43.874503 26010 consensus_queue.cc:1035] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bffeb10610b74a5b8737831dfb781eb6" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 34765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:43.881644 26015 mvcc.cc:205] Tried to move back new op lower bound from 7317047557619404800 to 7317047557489414144. Current Snapshot: MvccSnapshot[applied={T|T < 7317047557619404800}]
I20260810 19:02:43.890964 25297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:43.891098 25297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37628:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:43.893463 25297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 095cd2e9c0cc469ab50c8b78d0a1006d: Sending DeleteTablet for 3 replicas of tablet 4907e04fbbd74670885ef537f9c561a0
I20260810 19:02:43.894208 25676 tablet_service.cc:1514] Processing DeleteTablet for tablet 4907e04fbbd74670885ef537f9c561a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:43 UTC) from {username='slave'} at 127.0.0.1:34444
I20260810 19:02:43.894320 25867 tablet_service.cc:1514] Processing DeleteTablet for tablet 4907e04fbbd74670885ef537f9c561a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:43 UTC) from {username='slave'} at 127.0.0.1:60310
I20260810 19:02:43.894701 25481 tablet_service.cc:1514] Processing DeleteTablet for tablet 4907e04fbbd74670885ef537f9c561a0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:43 UTC) from {username='slave'} at 127.0.0.1:54982
I20260810 19:02:43.895046 26030 tablet_replica.cc:330] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: stopping tablet replica
I20260810 19:02:43.895349 26029 tablet_replica.cc:330] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: stopping tablet replica
I20260810 19:02:43.895594 26032 tablet_replica.cc:330] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2: stopping tablet replica
I20260810 19:02:43.895673 26029 raft_consensus.cc:2229] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:43.895704 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 25391
I20260810 19:02:43.895861 26032 raft_consensus.cc:2229] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:43.895969 26029 raft_consensus.cc:2258] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:43.896100 26032 raft_consensus.cc:2258] T 4907e04fbbd74670885ef537f9c561a0 P 2dc00b17497d4582a2387271af5444b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:43.896329 26030 raft_consensus.cc:2229] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:02:43.896721 26030 raft_consensus.cc:2258] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:43.901528 26030 ts_tablet_manager.cc:1905] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:43.903554 26029 ts_tablet_manager.cc:1905] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:43.904304 26030 ts_tablet_manager.cc:1918] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:43.904403 26030 log.cc:1196] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/4907e04fbbd74670885ef537f9c561a0
I20260810 19:02:43.904716 26030 ts_tablet_manager.cc:1939] T 4907e04fbbd74670885ef537f9c561a0 P 2686ffe8075a4398b2454dedd8d68c00: Deleting consensus metadata
I20260810 19:02:43.905809 25275 catalog_manager.cc:4841] TS 2686ffe8075a4398b2454dedd8d68c00 (127.27.147.131:39321): tablet 4907e04fbbd74670885ef537f9c561a0 (table test-table [id=69101cf6997245bdab0aba479a6f33ae]) successfully deleted
I20260810 19:02:43.907462 26029 ts_tablet_manager.cc:1918] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:43.907572 26029 log.cc:1196] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/4907e04fbbd74670885ef537f9c561a0
I20260810 19:02:43.907945 26029 ts_tablet_manager.cc:1939] T 4907e04fbbd74670885ef537f9c561a0 P bffeb10610b74a5b8737831dfb781eb6: Deleting consensus metadata
I20260810 19:02:43.909166 25276 catalog_manager.cc:4841] TS bffeb10610b74a5b8737831dfb781eb6 (127.27.147.130:34765): tablet 4907e04fbbd74670885ef537f9c561a0 (table test-table [id=69101cf6997245bdab0aba479a6f33ae]) successfully deleted
W20260810 19:02:43.912218 25277 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:37711 (error 108)
I20260810 19:02:43.913028 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 25582
I20260810 19:02:43.913266 25989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:43.912988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:37711 (local address 127.0.0.1:54986)
0810 19:02:43.913124 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:43.913190 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:37711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:02:43.920145 25277 catalog_manager.cc:4584] TS 2dc00b17497d4582a2387271af5444b2 (127.27.147.129:37711): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4907e04fbbd74670885ef537f9c561a0: Network error: Client connection negotiation failed: client connection to 127.27.147.129:37711: connect: Connection refused (error 111)
I20260810 19:02:43.925901 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 25776
I20260810 19:02:43.940415 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 25254
2026-08-10T19:02:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1180 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:44 dist-test-slave-znh6 krb5kdc[26039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:44 dist-test-slave-znh6 krb5kdc[26039](info): set up 2 sockets
Aug 10 19:02:44 dist-test-slave-znh6 krb5kdc[26039](info): commencing operation
krb5kdc: starting...
W20260810 19:02:45.185329 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.215s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:45 dist-test-slave-znh6 krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:02:45.310242 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38597
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:38597
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:45.419957 26056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:45.420176 26056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:45.420215 26056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:45.424149 26056 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:45.424197 26056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:45.424216 26056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:45.424232 26056 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:45.424271 26056 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:45.429265 26056 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:38597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:38597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26056
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:45.430205 26056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:45.431244 26056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:45.439744 26068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:45.439918 26056 server_base.cc:1034] running on GCE node
W20260810 19:02:45.439771 26064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:45.439831 26062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:45.440650 26056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:45.441637 26056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:45.442800 26056 hybrid_clock.cc:648] HybridClock initialized: now 1786388565442778 us; error 35 us; skew 500 ppm
Aug 10 19:02:45 dist-test-slave-znh6 krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388565, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:45.445644 26056 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:02:45.446434 26056 webserver.cc:460] Webserver started at http://127.27.147.190:38371/ using document root <none> and password file <none>
I20260810 19:02:45.447021 26056 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:45.447084 26056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:45.447248 26056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:45.449129 26056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "be0ccc8543aa4037913f81d6957b4fd1"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "604f4812a66b94825b1d90f9a14517bc"
server_key_iv: "4ffb726034032e53ab530e6e7e74d7c2"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.449625 26056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "be0ccc8543aa4037913f81d6957b4fd1"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "604f4812a66b94825b1d90f9a14517bc"
server_key_iv: "4ffb726034032e53ab530e6e7e74d7c2"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.453171 26056 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.004s
I20260810 19:02:45.455454 26079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:45.456602 26056 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:02:45.456705 26056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "be0ccc8543aa4037913f81d6957b4fd1"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "604f4812a66b94825b1d90f9a14517bc"
server_key_iv: "4ffb726034032e53ab530e6e7e74d7c2"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.456786 26056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:45.481431 26056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:45.484617 26056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:45.484772 26056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:45.491847 26056 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:38597
I20260810 19:02:45.491875 26159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:38597 every 8 connection(s)
I20260810 19:02:45.492784 26056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:45.495443 26160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:45.496045 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 26056
I20260810 19:02:45.496177 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:45.496474 28238 external_mini_cluster.cc:1428] Setting key 4a6562388c41bea87137bad38b6f3d96
I20260810 19:02:45.502239 26160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1: Bootstrap starting.
I20260810 19:02:45.504951 26160 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:45.505880 26160 log.cc:824] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1: Log is configured to *not* fsync() on all Append() calls
Aug 10 19:02:45 dist-test-slave-znh6 krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388565, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:45.507795 26160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1: No bootstrap required, opened a new log
I20260810 19:02:45.511991 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:45.497871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35338 (local address 127.27.147.190:38597)
0810 19:02:45.498290 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:45.498309 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:45.498354 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:02:45.499219 (+ 865us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:45.499244 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:45.499670 (+ 426us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:45.500025 (+ 355us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:45.500733 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.502232 (+ 1499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.503501 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.503953 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.504076 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:45.506891 (+ 2815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:45.506898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:45.506922 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:45.506950 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:45.508909 (+ 1959us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:45.509331 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:45.509337 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:45.509343 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:45.509495 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:45.509761 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:45.509765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:45.509768 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:45.510329 (+ 561us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:45.510545 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:45.510758 (+ 213us) server_negotiation.cc:299] Negotiation successful
0810 19:02:45.511067 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":110,"threads_started":1}
I20260810 19:02:45.512360 26160 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ccc8543aa4037913f81d6957b4fd1" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38597 } }
I20260810 19:02:45.512620 26160 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:45.512686 26160 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0ccc8543aa4037913f81d6957b4fd1, State: Initialized, Role: FOLLOWER
I20260810 19:02:45.513126 26160 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ccc8543aa4037913f81d6957b4fd1" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38597 } }
I20260810 19:02:45.513255 26160 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:45.513342 26160 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:45.513466 26160 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:45.514436 26160 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ccc8543aa4037913f81d6957b4fd1" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38597 } }
I20260810 19:02:45.514765 26160 leader_election.cc:302] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0ccc8543aa4037913f81d6957b4fd1; no voters:
I20260810 19:02:45.515115 26160 leader_election.cc:288] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:45.515232 26169 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:45.515508 26169 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [term 1 LEADER]: Becoming Leader. State: Replica: be0ccc8543aa4037913f81d6957b4fd1, State: Running, Role: LEADER
I20260810 19:02:45.515829 26169 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ccc8543aa4037913f81d6957b4fd1" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38597 } }
I20260810 19:02:45.516390 26160 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:45.517567 26172 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0ccc8543aa4037913f81d6957b4fd1. Latest consensus state: current_term: 1 leader_uuid: "be0ccc8543aa4037913f81d6957b4fd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ccc8543aa4037913f81d6957b4fd1" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38597 } } }
I20260810 19:02:45.517719 26172 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:45.518020 26171 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be0ccc8543aa4037913f81d6957b4fd1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0ccc8543aa4037913f81d6957b4fd1" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 38597 } } }
I20260810 19:02:45.518131 26171 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:45.518215 26181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:45.521373 26181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:45.526322 26181 catalog_manager.cc:1347] Generated new cluster ID: d09151e595074e1a81a82710a4744364
I20260810 19:02:45.526391 26181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:45.539484 26181 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:45.540446 26181 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:45.546983 26181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1: Generated new TSK 0
I20260810 19:02:45.547577 26181 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.27.147.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
Entry for principal kudu/127.27.147.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab.
I20260810 19:02:45.581398 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38597
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:45.693533 26201 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:45.693760 26201 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:45.693800 26201 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:45.697783 26201 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:45.697832 26201 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:45.697916 26201 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:02:45.702932 26201 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.129.keytab
--principal=kudu/127.27.147.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26201
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:45.703884 26201 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:45.704855 26201 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:45.714056 26208 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:45.714032 26206 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:45.714038 26212 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:45.714386 26201 server_base.cc:1034] running on GCE node
I20260810 19:02:45.714735 26201 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:45.715332 26201 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:45.716523 26201 hybrid_clock.cc:648] HybridClock initialized: now 1786388565716510 us; error 24 us; skew 500 ppm
Aug 10 19:02:45 dist-test-slave-znh6 krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388565, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:45.719497 26201 init.cc:377] Logged in from keytab as kudu/127.27.147.129@KRBTEST.COM (short username kudu)
I20260810 19:02:45.720342 26201 webserver.cc:460] Webserver started at http://127.27.147.129:34021/ using document root <none> and password file <none>
I20260810 19:02:45.720873 26201 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:45.720961 26201 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:45.721168 26201 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:45.723126 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "9c3228ced7874c97a6925584a3e39030"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "cf839a760b84be248f7cf1e6bc9d50fb"
server_key_iv: "edc9c48bcd0784edb1f64680182fb698"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.723694 26201 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "9c3228ced7874c97a6925584a3e39030"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "cf839a760b84be248f7cf1e6bc9d50fb"
server_key_iv: "edc9c48bcd0784edb1f64680182fb698"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.727170 26201 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:02:45.729417 26220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:45.730564 26201 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:45.730697 26201 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "9c3228ced7874c97a6925584a3e39030"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "cf839a760b84be248f7cf1e6bc9d50fb"
server_key_iv: "edc9c48bcd0784edb1f64680182fb698"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.730803 26201 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:45.756729 26201 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:45.759987 26201 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:45.760179 26201 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:45.760784 26201 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:45.761610 26201 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:45.761682 26201 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:45.761754 26201 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:45.761806 26201 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:45.770917 26201 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:45467
I20260810 19:02:45.770969 26398 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:45467 every 8 connection(s)
I20260810 19:02:45.771797 26201 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:02:45.777935 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 26201
I20260810 19:02:45.778079 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:02:45.778414 28238 external_mini_cluster.cc:1428] Setting key e5a9b05c21ae940ea556dbcc96b77ad1
Aug 10 19:02:45 dist-test-slave-znh6 krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388565, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.129@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:45.788595 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:45.773594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:42415 (local address 127.27.147.190:38597)
0810 19:02:45.773740 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:45.773744 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:45.776398 (+ 2654us) server_negotiation.cc:407] Connection header received
0810 19:02:45.777828 (+ 1430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:45.777833 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:45.777897 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:45.777999 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:45.779535 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.780144 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.781293 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.781700 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.781828 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:45.784406 (+ 2578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:45.784413 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:45.784418 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:45.784452 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:45.786223 (+ 1771us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:45.786759 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:45.786765 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:45.786769 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:45.786900 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:45.787313 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:45.787319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:45.787322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:45.787596 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:45.787720 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:45.788247 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 19:02:45.788393 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:02:45.789198 26401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:45.774008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38597 (local address 127.27.147.129:42415)
0810 19:02:45.776227 (+ 2219us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:45.776289 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:45.777606 (+ 1317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:45.778221 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:45.778229 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:45.778686 (+ 457us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:45.779378 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:45.779398 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.780275 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.780278 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:45.781149 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:45.781159 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.781850 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.781852 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:45.782033 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:45.782629 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:45.782645 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:45.784243 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:45.786359 (+ 2116us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:45.786363 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:45.786375 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:45.786627 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:45.787019 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:45.787022 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:45.787024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:45.787177 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:45.787731 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:45.787737 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:45.787979 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 19:02:45.788126 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 19:02:45.788359 (+ 233us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1993,"thread_start_us":1829,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.130@KRBTEST.COM; defaulting to no policy
I20260810 19:02:45.790557 26399 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38597
I20260810 19:02:45.790843 26399 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:45.791318 26399 heartbeater.cc:511] Master 127.27.147.190:38597 requested a full tablet report, sending...
I20260810 19:02:45.792832 26102 ts_manager.cc:194] Registered new tserver with Master: 9c3228ced7874c97a6925584a3e39030 (127.27.147.129:45467)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.130@KRBTEST.COM" created.
I20260810 19:02:45.794111 26102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.129@KRBTEST.COM'} at 127.27.147.129:42415
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
Entry for principal kudu/127.27.147.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab.
I20260810 19:02:45.809258 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38597
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:45.920209 26405 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:45.920501 26405 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:45.920580 26405 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:45.925011 26405 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:45.925065 26405 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:45.925119 26405 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:02:45.930058 26405 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.130.keytab
--principal=kudu/127.27.147.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26405
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:45.930949 26405 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:45.931816 26405 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:45.941313 26410 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:45.941308 26413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:45.941308 26411 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:45.941603 26405 server_base.cc:1034] running on GCE node
I20260810 19:02:45.942010 26405 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:45.942606 26405 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:45.943768 26405 hybrid_clock.cc:648] HybridClock initialized: now 1786388565943754 us; error 33 us; skew 500 ppm
Aug 10 19:02:45 dist-test-slave-znh6 krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388565, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:45.946533 26405 init.cc:377] Logged in from keytab as kudu/127.27.147.130@KRBTEST.COM (short username kudu)
I20260810 19:02:45.947407 26405 webserver.cc:460] Webserver started at http://127.27.147.130:46283/ using document root <none> and password file <none>
I20260810 19:02:45.947939 26405 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:45.947999 26405 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:45.948204 26405 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:45.950124 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "1c1c5f01a999b104e9f1cff07e0919f8"
server_key_iv: "ab3c23c0b4057d3fdc449ae6e7e9adda"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.950672 26405 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "1c1c5f01a999b104e9f1cff07e0919f8"
server_key_iv: "ab3c23c0b4057d3fdc449ae6e7e9adda"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.954134 26405 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260810 19:02:45.956588 26425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:45.957737 26405 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:45.957859 26405 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc"
format_stamp: "Formatted at 2026-08-10 19:02:45 on dist-test-slave-znh6"
server_key: "1c1c5f01a999b104e9f1cff07e0919f8"
server_key_iv: "ab3c23c0b4057d3fdc449ae6e7e9adda"
server_key_version: "encryptionkey@0"
I20260810 19:02:45.957960 26405 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:45.968178 26405 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:45.970731 26405 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:45.970948 26405 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:45.971495 26405 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:45.972281 26405 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:45.972329 26405 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:45.972396 26405 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:45.972435 26405 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:45.981146 26405 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:40191
I20260810 19:02:45.981168 26598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:40191 every 8 connection(s)
I20260810 19:02:45.981993 26405 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:02:45.985146 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 26405
I20260810 19:02:45.985332 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:02:45.985666 28238 external_mini_cluster.cc:1428] Setting key 3636752b83b39b2ec3dbe5da542333d2
Aug 10 19:02:45 dist-test-slave-znh6 krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388565, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.130@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:45.996455 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:45.983815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:45449 (local address 127.27.147.190:38597)
0810 19:02:45.983953 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:45.983965 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:45.984747 (+ 782us) server_negotiation.cc:407] Connection header received
0810 19:02:45.986154 (+ 1407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:45.986157 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:45.986217 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:45.986323 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:45.987591 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.988116 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.989405 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.989631 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.989717 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:45.992566 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:45.992571 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:45.992576 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:45.992609 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:45.994371 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:45.994874 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:45.994879 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:45.994883 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:45.994985 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:45.995312 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:45.995317 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:45.995319 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:45.995561 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:45.995677 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:45.996124 (+ 447us) server_negotiation.cc:299] Negotiation successful
0810 19:02:45.996256 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:02:45.997068 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:45.984156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38597 (local address 127.27.147.130:45449)
0810 19:02:45.984595 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:45.984636 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:45.985941 (+ 1305us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:45.986506 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:45.986513 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:45.986954 (+ 441us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:45.987450 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:45.987464 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.988248 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.988251 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:45.989273 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:45.989287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:45.989788 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:45.989795 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:45.990015 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:45.990630 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:45.990659 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:45.992390 (+ 1731us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:45.994491 (+ 2101us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:45.994495 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:45.994506 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:45.994750 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:45.995085 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:45.995087 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:45.995089 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:45.995216 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:45.995670 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:45.995675 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:45.995890 (+ 215us) client_negotiation.cc:769] Sending connection context
0810 19:02:45.996037 (+ 147us) client_negotiation.cc:240] Negotiation successful
0810 19:02:45.996262 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":262,"thread_start_us":114,"threads_started":1}
WARNING: no policy specified for kudu/127.27.147.131@KRBTEST.COM; defaulting to no policy
I20260810 19:02:45.998165 26601 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38597
I20260810 19:02:45.998399 26601 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:45.998921 26601 heartbeater.cc:511] Master 127.27.147.190:38597 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.131@KRBTEST.COM" created.
I20260810 19:02:46.000034 26102 ts_manager.cc:194] Registered new tserver with Master: 5d1fe3b4e02c4c83921f8d84dff4c0fc (127.27.147.130:40191)
I20260810 19:02:46.000589 26102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.130@KRBTEST.COM'} at 127.27.147.130:45449
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
Entry for principal kudu/127.27.147.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab.
I20260810 19:02:46.015744 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38597
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:46.127775 26610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:46.128036 26610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:46.128120 26610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:46.132078 26610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:46.132154 26610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:46.132236 26610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:46.137573 26610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.131.keytab
--principal=kudu/127.27.147.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:38597
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:46.138556 26610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:46.139540 26610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:46.149220 26618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:46.149230 26616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:46.149237 26620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:46.149487 26610 server_base.cc:1034] running on GCE node
I20260810 19:02:46.149961 26610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:46.150624 26610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:46.151784 26610 hybrid_clock.cc:648] HybridClock initialized: now 1786388566151777 us; error 59 us; skew 500 ppm
Aug 10 19:02:46 dist-test-slave-znh6 krb5kdc[26039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:46.154554 26610 init.cc:377] Logged in from keytab as kudu/127.27.147.131@KRBTEST.COM (short username kudu)
I20260810 19:02:46.155509 26610 webserver.cc:460] Webserver started at http://127.27.147.131:45141/ using document root <none> and password file <none>
I20260810 19:02:46.156111 26610 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:46.156204 26610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:46.156450 26610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:46.158473 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "ce34e258754c42e1b0da063ddcb86e57"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "f2681d18bbb6cd507f648e4f2bb4019e"
server_key_iv: "2d1d2f6f9048fa0f3d7b1bb82b02d505"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.159075 26610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "ce34e258754c42e1b0da063ddcb86e57"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "f2681d18bbb6cd507f648e4f2bb4019e"
server_key_iv: "2d1d2f6f9048fa0f3d7b1bb82b02d505"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.163038 26610 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 19:02:46.165359 26631 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:46.166596 26610 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:46.166735 26610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "ce34e258754c42e1b0da063ddcb86e57"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "f2681d18bbb6cd507f648e4f2bb4019e"
server_key_iv: "2d1d2f6f9048fa0f3d7b1bb82b02d505"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.166868 26610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:46.185451 26610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:46.188254 26610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:46.188455 26610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:46.189024 26610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:46.189821 26610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:46.189900 26610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:46.189973 26610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:46.190030 26610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:46.199142 26610 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:34943
I20260810 19:02:46.199175 26803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:34943 every 8 connection(s)
I20260810 19:02:46.200001 26610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:02:46.202044 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 26610
I20260810 19:02:46.202183 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:02:46.202500 28238 external_mini_cluster.cc:1428] Setting key d8423732919ce77a554ea465019e2bb4
Aug 10 19:02:46 dist-test-slave-znh6 krb5kdc[26039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388566, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.131@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:02:46.213414 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.201755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:48933 (local address 127.27.147.190:38597)
0810 19:02:46.201939 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.201945 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.202840 (+ 895us) server_negotiation.cc:407] Connection header received
0810 19:02:46.204456 (+ 1616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.204459 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.204513 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.204601 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:46.205819 (+ 1218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.206360 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.207528 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.207742 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.207826 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.210082 (+ 2256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:46.210091 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:46.210094 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:46.210116 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:46.211438 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:46.211934 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:46.211939 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:46.211941 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:46.212028 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:46.212356 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:46.212360 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:46.212362 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:46.212532 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:46.212640 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.213097 (+ 457us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.213238 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 19:02:46.214102 26806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.202143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:38597 (local address 127.27.147.131:48933)
0810 19:02:46.202654 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.202706 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:46.204253 (+ 1547us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:46.204805 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:46.204816 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:46.205230 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:46.205686 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.205699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.206469 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.206472 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.207413 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.207423 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.207877 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.207880 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.208013 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.208484 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:02:46.208499 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:46.209927 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:46.211558 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:46.211563 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:46.211576 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:46.211825 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:46.212122 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:02:46.212126 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:02:46.212128 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:02:46.212260 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:02:46.212640 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:46.212647 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:46.212881 (+ 234us) client_negotiation.cc:769] Sending connection context
0810 19:02:46.213057 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 19:02:46.213363 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305,"thread_start_us":132,"threads_started":1}
I20260810 19:02:46.215306 26804 heartbeater.cc:347] Connected to a master server at 127.27.147.190:38597
I20260810 19:02:46.215543 26804 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:46.216215 26804 heartbeater.cc:511] Master 127.27.147.190:38597 requested a full tablet report, sending...
I20260810 19:02:46.217310 26102 ts_manager.cc:194] Registered new tserver with Master: ce34e258754c42e1b0da063ddcb86e57 (127.27.147.131:34943)
I20260810 19:02:46.217860 26102 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.27.147.131@KRBTEST.COM'} at 127.27.147.131:48933
I20260810 19:02:46.228222 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:02:46.237176 26165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.230144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.27.147.190:38597)
0810 19:02:46.230282 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.230286 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.230419 (+ 133us) server_negotiation.cc:407] Connection header received
0810 19:02:46.230514 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.230516 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.230552 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.230615 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:46.231338 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.231825 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.232598 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.232849 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.232956 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.234195 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:46.234198 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:46.234200 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:46.234221 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:46.235761 (+ 1540us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:46.236127 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:46.236129 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:46.236131 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:46.236201 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:46.236429 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:46.236431 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:46.236432 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:46.236618 (+ 186us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:46.236727 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.236916 (+ 189us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.237042 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:02:46.239820 26102 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:02:46.242782 26102 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:02:46.256800 26819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.250961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:40191 (local address 127.0.0.1:45504)
0810 19:02:46.251576 (+ 615us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.251623 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:46.251757 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:46.252598 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:46.252601 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:46.252619 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:46.252776 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.252782 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.253904 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.253917 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.255230 (+ 1313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.255241 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.256215 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.256219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.256326 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.256362 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 19:02:46.256506 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 19:02:46.256621 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"mutex_wait_us":3,"spinlock_wait_cycles":3328,"thread_start_us":100,"threads_started":1}
I20260810 19:02:46.257139 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.250951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51828 (local address 127.27.147.131:34943)
0810 19:02:46.252316 (+ 1365us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.252334 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.252353 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:02:46.252413 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.252416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.252603 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.252793 (+ 190us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:46.253028 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.254124 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.255557 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.256390 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.256521 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.256653 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.256879 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.256994 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1267,"thread_start_us":138,"threads_started":1}
I20260810 19:02:46.257141 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.251226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:45467 (local address 127.0.0.1:37494)
0810 19:02:46.251978 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.251990 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:46.252072 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:46.252542 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:46.252547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:46.252568 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:46.252743 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.252752 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.254448 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.254454 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.255642 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.255653 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.256738 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.256742 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.256829 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.256852 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 19:02:46.256940 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 19:02:46.257005 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":697,"mutex_wait_us":28,"thread_start_us":81,"threads_started":1}
I20260810 19:02:46.257400 26817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.250820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:34943 (local address 127.0.0.1:51828)
0810 19:02:46.251574 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.251621 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:46.251756 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:46.252789 (+ 1033us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:46.252792 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:46.252811 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:46.252904 (+ 93us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.252919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.254258 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.254262 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.255438 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.255447 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.256519 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.256522 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.256614 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.256636 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 19:02:46.256731 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 19:02:46.256789 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":46,"spinlock_wait_cycles":6912,"thread_start_us":81,"threads_started":1}
I20260810 19:02:46.257464 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.251781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45504 (local address 127.27.147.130:40191)
0810 19:02:46.252186 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.252196 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.252215 (+ 19us) server_negotiation.cc:407] Connection header received
0810 19:02:46.252279 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.252283 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.252450 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.252932 (+ 482us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:46.253088 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.253775 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.255371 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.256089 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.256228 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.256360 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.257221 (+ 861us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.257333 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":104,"threads_started":1}
I20260810 19:02:46.258365 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.251440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37494 (local address 127.27.147.129:45467)
0810 19:02:46.252055 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.252073 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.252099 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:02:46.252179 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.252184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.252377 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.252546 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:46.252875 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.254109 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.255770 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.256601 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.256725 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.256843 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.257117 (+ 274us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.258233 (+ 1116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":104,"threads_started":1}
I20260810 19:02:46.260139 26494 tablet_service.cc:1467] Processing CreateTablet for tablet 076bbd806a6d49e2bea1c6ad9a5d45de (DEFAULT_TABLE table=test-table [id=2330fc03b6834843a0a8de699fe7d3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:46.260231 26697 tablet_service.cc:1467] Processing CreateTablet for tablet 076bbd806a6d49e2bea1c6ad9a5d45de (DEFAULT_TABLE table=test-table [id=2330fc03b6834843a0a8de699fe7d3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:46.261268 26697 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076bbd806a6d49e2bea1c6ad9a5d45de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:46.261271 26494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076bbd806a6d49e2bea1c6ad9a5d45de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:46.260949 26301 tablet_service.cc:1467] Processing CreateTablet for tablet 076bbd806a6d49e2bea1c6ad9a5d45de (DEFAULT_TABLE table=test-table [id=2330fc03b6834843a0a8de699fe7d3ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:46.261873 26301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 076bbd806a6d49e2bea1c6ad9a5d45de. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:46.267493 26824 tablet_bootstrap.cc:492] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Bootstrap starting.
I20260810 19:02:46.269263 26825 tablet_bootstrap.cc:492] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Bootstrap starting.
I20260810 19:02:46.270129 26824 tablet_bootstrap.cc:654] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:46.270949 26826 tablet_bootstrap.cc:492] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030: Bootstrap starting.
I20260810 19:02:46.271513 26825 tablet_bootstrap.cc:654] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:46.271493 26824 log.cc:824] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:46.272804 26825 log.cc:824] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:46.273092 26826 tablet_bootstrap.cc:654] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:46.273957 26824 tablet_bootstrap.cc:492] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: No bootstrap required, opened a new log
I20260810 19:02:46.274152 26824 ts_tablet_manager.cc:1397] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.000s
I20260810 19:02:46.274287 26826 log.cc:824] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:46.274606 26825 tablet_bootstrap.cc:492] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: No bootstrap required, opened a new log
I20260810 19:02:46.274838 26825 ts_tablet_manager.cc:1397] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.001s
I20260810 19:02:46.276249 26826 tablet_bootstrap.cc:492] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030: No bootstrap required, opened a new log
I20260810 19:02:46.276468 26826 ts_tablet_manager.cc:1397] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260810 19:02:46.277521 26825 raft_consensus.cc:348] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.277750 26825 raft_consensus.cc:374] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:46.277804 26825 raft_consensus.cc:729] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce34e258754c42e1b0da063ddcb86e57, State: Initialized, Role: FOLLOWER
I20260810 19:02:46.278108 26824 raft_consensus.cc:348] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.278235 26825 consensus_queue.cc:260] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.278350 26824 raft_consensus.cc:374] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:46.278441 26824 raft_consensus.cc:729] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d1fe3b4e02c4c83921f8d84dff4c0fc, State: Initialized, Role: FOLLOWER
I20260810 19:02:46.278856 26824 consensus_queue.cc:260] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.279307 26825 ts_tablet_manager.cc:1428] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Time spent starting tablet: real 0.004s user 0.001s sys 0.003s
I20260810 19:02:46.279682 26804 heartbeater.cc:503] Master 127.27.147.190:38597 was elected leader, sending a full tablet report...
I20260810 19:02:46.279757 26824 ts_tablet_manager.cc:1428] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 19:02:46.280077 26601 heartbeater.cc:503] Master 127.27.147.190:38597 was elected leader, sending a full tablet report...
I20260810 19:02:46.280179 26826 raft_consensus.cc:348] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.280342 26826 raft_consensus.cc:374] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:46.280370 26826 raft_consensus.cc:729] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c3228ced7874c97a6925584a3e39030, State: Initialized, Role: FOLLOWER
I20260810 19:02:46.280742 26826 consensus_queue.cc:260] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.281356 26399 heartbeater.cc:503] Master 127.27.147.190:38597 was elected leader, sending a full tablet report...
I20260810 19:02:46.282763 26826 ts_tablet_manager.cc:1428] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:02:46.312422 26836 raft_consensus.cc:482] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:46.312602 26836 raft_consensus.cc:504] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.313714 26836 leader_election.cc:288] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce34e258754c42e1b0da063ddcb86e57 (127.27.147.131:34943), 9c3228ced7874c97a6925584a3e39030 (127.27.147.129:45467)
I20260810 19:02:46.318113 26822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.314063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:37531 (local address 127.27.147.129:45467)
0810 19:02:46.314188 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.314195 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.314452 (+ 257us) server_negotiation.cc:407] Connection header received
0810 19:02:46.314543 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.314546 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.314592 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.314651 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:46.314932 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.315640 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.317013 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.317554 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.317682 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.317738 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.317937 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.318010 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:02:46.318131 26839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.313982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:45467 (local address 127.27.147.130:37531)
0810 19:02:46.314334 (+ 352us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.314344 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:46.314459 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:46.314694 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:46.314698 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:46.314719 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:46.314842 (+ 123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.314847 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.315740 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.315742 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.316886 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.316897 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.317673 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.317677 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.317805 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.317825 (+ 20us) client_negotiation.cc:769] Sending connection context
0810 19:02:46.317914 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 19:02:46.317981 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":135,"thread_start_us":111,"threads_started":1}
I20260810 19:02:46.318183 26820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.314090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:49873 (local address 127.27.147.131:34943)
0810 19:02:46.314227 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.314231 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.314247 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:02:46.314299 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.314301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.314338 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.314431 (+ 93us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:46.314741 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.315579 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.317037 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.317619 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.317744 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.317797 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.318002 (+ 205us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.318062 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:02:46.318423 26604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.313871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:34943 (local address 127.27.147.130:49873)
0810 19:02:46.314013 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.314029 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:46.314117 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:46.314449 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:46.314452 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:46.314466 (+ 14us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:46.314605 (+ 139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.314610 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.315709 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.315714 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.316909 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.316922 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.317744 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.317748 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.317875 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.317900 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 19:02:46.317986 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:02:46.318057 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56,"mutex_wait_us":152}
I20260810 19:02:46.318941 26332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd806a6d49e2bea1c6ad9a5d45de" candidate_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c3228ced7874c97a6925584a3e39030" is_pre_election: true
I20260810 19:02:46.319007 26725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd806a6d49e2bea1c6ad9a5d45de" candidate_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce34e258754c42e1b0da063ddcb86e57" is_pre_election: true
I20260810 19:02:46.319204 26332 raft_consensus.cc:2454] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1fe3b4e02c4c83921f8d84dff4c0fc in term 0.
I20260810 19:02:46.319330 26725 raft_consensus.cc:2454] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d1fe3b4e02c4c83921f8d84dff4c0fc in term 0.
I20260810 19:02:46.319674 26429 leader_election.cc:302] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1fe3b4e02c4c83921f8d84dff4c0fc, 9c3228ced7874c97a6925584a3e39030; no voters:
I20260810 19:02:46.319954 26836 raft_consensus.cc:2781] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:02:46.320041 26836 raft_consensus.cc:482] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:02:46.320086 26836 raft_consensus.cc:3037] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:46.321121 26836 raft_consensus.cc:504] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.321427 26836 leader_election.cc:288] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [CANDIDATE]: Term 1 election: Requested vote from peers ce34e258754c42e1b0da063ddcb86e57 (127.27.147.131:34943), 9c3228ced7874c97a6925584a3e39030 (127.27.147.129:45467)
I20260810 19:02:46.321717 26725 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd806a6d49e2bea1c6ad9a5d45de" candidate_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce34e258754c42e1b0da063ddcb86e57"
I20260810 19:02:46.321759 26332 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "076bbd806a6d49e2bea1c6ad9a5d45de" candidate_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c3228ced7874c97a6925584a3e39030"
I20260810 19:02:46.321841 26725 raft_consensus.cc:3037] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:46.321866 26332 raft_consensus.cc:3037] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:46.323151 26725 raft_consensus.cc:2454] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1fe3b4e02c4c83921f8d84dff4c0fc in term 1.
I20260810 19:02:46.323151 26332 raft_consensus.cc:2454] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d1fe3b4e02c4c83921f8d84dff4c0fc in term 1.
I20260810 19:02:46.323513 26430 leader_election.cc:302] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d1fe3b4e02c4c83921f8d84dff4c0fc, ce34e258754c42e1b0da063ddcb86e57; no voters:
I20260810 19:02:46.323704 26836 raft_consensus.cc:2781] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:46.323902 26836 raft_consensus.cc:686] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 1 LEADER]: Becoming Leader. State: Replica: 5d1fe3b4e02c4c83921f8d84dff4c0fc, State: Running, Role: LEADER
I20260810 19:02:46.324221 26836 consensus_queue.cc:237] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } }
I20260810 19:02:46.327615 26102 catalog_manager.cc:5495] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d1fe3b4e02c4c83921f8d84dff4c0fc (127.27.147.130). New cstate: current_term: 1 leader_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d1fe3b4e02c4c83921f8d84dff4c0fc" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 40191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 } health_report { overall_health: UNKNOWN } } }
I20260810 19:02:46.351761 26823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.348684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.27.147.130:40191)
0810 19:02:46.348785 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.348789 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.348805 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:02:46.348955 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.348958 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.349006 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.349088 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:46.349298 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.349769 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.350545 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.350740 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.350859 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.351125 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:46.351484 (+ 359us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:02:46.351570 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.351626 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.351670 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 19:02:46.357074 26725 raft_consensus.cc:1261] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 1 FOLLOWER]: Refusing update from remote peer 5d1fe3b4e02c4c83921f8d84dff4c0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:46.357074 26332 raft_consensus.cc:1261] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 1 FOLLOWER]: Refusing update from remote peer 5d1fe3b4e02c4c83921f8d84dff4c0fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:46.357726 26836 consensus_queue.cc:1035] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c3228ced7874c97a6925584a3e39030" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 45467 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:46.357957 26841 consensus_queue.cc:1035] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce34e258754c42e1b0da063ddcb86e57" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 34943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:46.366111 26846 mvcc.cc:205] Tried to move back new op lower bound from 7317047567791968256 to 7317047567667056640. Current Snapshot: MvccSnapshot[applied={T|T < 7317047567791968256}]
I20260810 19:02:46.374094 26102 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:46.374264 26102 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:46.376682 26102 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be0ccc8543aa4037913f81d6957b4fd1: Sending DeleteTablet for 3 replicas of tablet 076bbd806a6d49e2bea1c6ad9a5d45de
I20260810 19:02:46.377461 26301 tablet_service.cc:1514] Processing DeleteTablet for tablet 076bbd806a6d49e2bea1c6ad9a5d45de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:46 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:37494
I20260810 19:02:46.377465 26697 tablet_service.cc:1514] Processing DeleteTablet for tablet 076bbd806a6d49e2bea1c6ad9a5d45de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:46 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:51828
I20260810 19:02:46.377543 26494 tablet_service.cc:1514] Processing DeleteTablet for tablet 076bbd806a6d49e2bea1c6ad9a5d45de with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:46 UTC) from {username='kudu', principal='kudu/127.27.147.190@KRBTEST.COM'} at 127.0.0.1:45504
I20260810 19:02:46.378005 26861 tablet_replica.cc:330] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030: stopping tablet replica
I20260810 19:02:46.378072 26863 tablet_replica.cc:330] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: stopping tablet replica
I20260810 19:02:46.378376 26863 raft_consensus.cc:2229] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 1 LEADER]: Raft consensus shutting down.
I20260810 19:02:46.378490 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 26201
I20260810 19:02:46.378391 26861 raft_consensus.cc:2229] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:46.378760 26863 raft_consensus.cc:2258] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:46.378908 26861 raft_consensus.cc:2258] T 076bbd806a6d49e2bea1c6ad9a5d45de P 9c3228ced7874c97a6925584a3e39030 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:46.379194 26862 tablet_replica.cc:330] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: stopping tablet replica
I20260810 19:02:46.380769 26863 ts_tablet_manager.cc:1905] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:46.381940 26862 raft_consensus.cc:2229] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:46.382284 26862 raft_consensus.cc:2258] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:46.383720 26863 ts_tablet_manager.cc:1918] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:46.383824 26863 log.cc:1196] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/076bbd806a6d49e2bea1c6ad9a5d45de
I20260810 19:02:46.384127 26862 ts_tablet_manager.cc:1905] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:46.384174 26863 ts_tablet_manager.cc:1939] T 076bbd806a6d49e2bea1c6ad9a5d45de P 5d1fe3b4e02c4c83921f8d84dff4c0fc: Deleting consensus metadata
I20260810 19:02:46.387609 26080 catalog_manager.cc:4841] TS 5d1fe3b4e02c4c83921f8d84dff4c0fc (127.27.147.130:40191): tablet 076bbd806a6d49e2bea1c6ad9a5d45de (table test-table [id=2330fc03b6834843a0a8de699fe7d3ac]) successfully deleted
I20260810 19:02:46.387845 26862 ts_tablet_manager.cc:1918] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:46.387975 26862 log.cc:1196] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/076bbd806a6d49e2bea1c6ad9a5d45de
I20260810 19:02:46.388322 26862 ts_tablet_manager.cc:1939] T 076bbd806a6d49e2bea1c6ad9a5d45de P ce34e258754c42e1b0da063ddcb86e57: Deleting consensus metadata
I20260810 19:02:46.389370 26083 catalog_manager.cc:4841] TS ce34e258754c42e1b0da063ddcb86e57 (127.27.147.131:34943): tablet 076bbd806a6d49e2bea1c6ad9a5d45de (table test-table [id=2330fc03b6834843a0a8de699fe7d3ac]) successfully deleted
W20260810 19:02:46.396456 26082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:45467 (error 108)
I20260810 19:02:46.396968 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 26405
I20260810 19:02:46.399096 26821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.398793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:45467 (local address 127.0.0.1:37500)
0810 19:02:46.398954 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.399022 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:45467: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260810 19:02:46.399328 26082 catalog_manager.cc:4584] TS 9c3228ced7874c97a6925584a3e39030 (127.27.147.129:45467): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 076bbd806a6d49e2bea1c6ad9a5d45de: Network error: Client connection negotiation failed: client connection to 127.27.147.129:45467: connect: Connection refused (error 111)
I20260810 19:02:46.410722 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 26610
I20260810 19:02:46.421398 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 26056
2026-08-10T19:02:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2481 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T19:02:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:46Z Disabled control of system clock
I20260810 19:02:46.460587 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:43411
--webserver_interface=127.27.147.190
--webserver_port=0
--builtin_ntp_servers=127.27.147.148:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:43411
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:02:46.569759 26868 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:46.570003 26868 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:46.570036 26868 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:46.573899 26868 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:46.573951 26868 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:46.573966 26868 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:46.573982 26868 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:46.573997 26868 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:46.578603 26868 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:43411
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:43411
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26868
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:46.579627 26868 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:46.580967 26868 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:46.589313 26873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:46.589397 26868 server_base.cc:1034] running on GCE node
W20260810 19:02:46.589296 26878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:46.589316 26875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:46.590193 26868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:46.591249 26868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:46.592407 26868 hybrid_clock.cc:648] HybridClock initialized: now 1786388566592387 us; error 42 us; skew 500 ppm
I20260810 19:02:46.594092 26868 webserver.cc:460] Webserver started at http://127.27.147.190:35541/ using document root <none> and password file <none>
I20260810 19:02:46.594601 26868 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:46.594659 26868 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:46.594870 26868 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:46.596841 26868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "0361b68f96584723a54480c115969e74"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "6c9d709278a525d70adc81e70e044656"
server_key_iv: "82d9416f2cb698d29b3bd5eed18521b2"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.597334 26868 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "0361b68f96584723a54480c115969e74"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "6c9d709278a525d70adc81e70e044656"
server_key_iv: "82d9416f2cb698d29b3bd5eed18521b2"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.600819 26868 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:02:46.603142 26886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:46.604292 26868 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:02:46.604393 26868 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "0361b68f96584723a54480c115969e74"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "6c9d709278a525d70adc81e70e044656"
server_key_iv: "82d9416f2cb698d29b3bd5eed18521b2"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.604480 26868 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:46.614405 26868 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:46.615027 26868 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:46.615160 26868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:46.621790 26868 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:43411
I20260810 19:02:46.621810 26971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:43411 every 8 connection(s)
I20260810 19:02:46.622648 26868 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:46.625701 26972 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:46.626062 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 26868
I20260810 19:02:46.626199 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:46.626507 28238 external_mini_cluster.cc:1428] Setting key 46b75ab8528f0ffd20f6abcd242e6c7c
I20260810 19:02:46.631481 26972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74: Bootstrap starting.
I20260810 19:02:46.633566 26972 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:46.634311 26972 log.cc:824] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:46.635553 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.627938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41330 (local address 127.27.147.190:43411)
0810 19:02:46.628535 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.628550 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.628584 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:02:46.629261 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.629279 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.629646 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.629976 (+ 330us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:46.630369 (+ 393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.631378 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.632296 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.632749 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.632930 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.633757 (+ 827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:46.633763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:46.633775 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:46.634150 (+ 375us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:46.634203 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:46.634223 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:46.634332 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.634452 (+ 120us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.634719 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":128,"threads_started":1}
I20260810 19:02:46.636303 26972 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74: No bootstrap required, opened a new log
I20260810 19:02:46.639766 26972 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361b68f96584723a54480c115969e74" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43411 } }
I20260810 19:02:46.640003 26972 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:46.640086 26972 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0361b68f96584723a54480c115969e74, State: Initialized, Role: FOLLOWER
I20260810 19:02:46.640494 26972 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361b68f96584723a54480c115969e74" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43411 } }
I20260810 19:02:46.640645 26972 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:46.640738 26972 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:46.640858 26972 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:46.641708 26972 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361b68f96584723a54480c115969e74" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43411 } }
I20260810 19:02:46.642041 26972 leader_election.cc:302] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0361b68f96584723a54480c115969e74; no voters:
I20260810 19:02:46.642364 26972 leader_election.cc:288] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:46.642493 26982 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:46.642745 26982 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [term 1 LEADER]: Becoming Leader. State: Replica: 0361b68f96584723a54480c115969e74, State: Running, Role: LEADER
I20260810 19:02:46.643047 26982 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361b68f96584723a54480c115969e74" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43411 } }
I20260810 19:02:46.643648 26972 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:46.644785 26984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0361b68f96584723a54480c115969e74. Latest consensus state: current_term: 1 leader_uuid: "0361b68f96584723a54480c115969e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361b68f96584723a54480c115969e74" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43411 } } }
I20260810 19:02:46.645066 26984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:46.644817 26983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0361b68f96584723a54480c115969e74" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0361b68f96584723a54480c115969e74" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43411 } } }
I20260810 19:02:46.645359 26983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:46.645644 26994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:46.648658 26994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:46.653982 26994 catalog_manager.cc:1347] Generated new cluster ID: f4f67d03f1d742ec807c645d0e219d34
I20260810 19:02:46.654045 26994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:46.661837 26994 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:46.662671 26994 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:46.675849 26994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74: Generated new TSK 0
I20260810 19:02:46.676479 26994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:46.680645 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.129:0
--local_ip_for_outbound_sockets=127.27.147.129
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:43411
--builtin_ntp_servers=127.27.147.148:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 19:02:46.791826 27010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:46.792049 27010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:46.792116 27010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:46.796061 27010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:46.796106 27010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:46.796200 27010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.129
I20260810 19:02:46.800792 27010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.27.147.129
--webserver_port=0
--tserver_master_addrs=127.27.147.190:43411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.129
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:46.801759 27010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:46.803115 27010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:02:46.812358 27010 server_base.cc:1034] running on GCE node
W20260810 19:02:46.812301 27017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:46.812333 27020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:46.812314 27016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:46.813000 27010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:46.813611 27010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:46.814795 27010 hybrid_clock.cc:648] HybridClock initialized: now 1786388566814765 us; error 51 us; skew 500 ppm
I20260810 19:02:46.820686 27010 webserver.cc:460] Webserver started at http://127.27.147.129:33365/ using document root <none> and password file <none>
I20260810 19:02:46.821246 27010 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:46.821339 27010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:46.821573 27010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:46.823565 27010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data/instance:
uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "a1a18af1be140a267f1e5bff0045d945"
server_key_iv: "08541cfda29686b6d9120abf097ec33e"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.824115 27010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance:
uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "a1a18af1be140a267f1e5bff0045d945"
server_key_iv: "08541cfda29686b6d9120abf097ec33e"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.827648 27010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:02:46.829847 27028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:46.831028 27010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.001s
I20260810 19:02:46.831158 27010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/wal
uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "a1a18af1be140a267f1e5bff0045d945"
server_key_iv: "08541cfda29686b6d9120abf097ec33e"
server_key_version: "encryptionkey@0"
I20260810 19:02:46.831262 27010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:46.840395 27010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:46.840989 27010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:46.841168 27010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:46.841681 27010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:46.842550 27010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:46.842620 27010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:46.842686 27010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:46.842741 27010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:46.851600 27010 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.129:39567
I20260810 19:02:46.851701 27206 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.129:39567 every 8 connection(s)
I20260810 19:02:46.852414 27010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/data/info.pb
I20260810 19:02:46.857519 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 27010
I20260810 19:02:46.857646 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-0/wal/instance
I20260810 19:02:46.857965 28238 external_mini_cluster.cc:1428] Setting key 8b8ba0db943e200c553471d52a6ff36f
I20260810 19:02:46.860023 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.130:0
--local_ip_for_outbound_sockets=127.27.147.130
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:43411
--builtin_ntp_servers=127.27.147.148:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:02:46.862021 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.854116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:39507 (local address 127.27.147.190:43411)
0810 19:02:46.854273 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:46.854278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:46.855292 (+ 1014us) server_negotiation.cc:407] Connection header received
0810 19:02:46.856634 (+ 1342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:46.856637 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:46.856683 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:46.856760 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:46.857569 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.858373 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.859733 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.860104 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.860230 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.861412 (+ 1182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:46.861419 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:46.861425 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:46.861442 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:46.861470 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:46.861483 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:46.861567 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:46.861738 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 19:02:46.861813 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 19:02:46.863106 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:46.854463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:43411 (local address 127.27.147.129:39507)
0810 19:02:46.855153 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:46.855192 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:46.856452 (+ 1260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:46.856894 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:46.856900 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:46.856977 (+ 77us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:46.857414 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.857426 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.858510 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.858514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.859603 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:46.859614 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:46.860405 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:46.860408 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:46.860543 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:46.861134 (+ 591us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:46.861135 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:46.861172 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:46.861230 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:46.861585 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:46.861593 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:46.861598 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 19:02:46.861700 (+ 102us) client_negotiation.cc:240] Negotiation successful
0810 19:02:46.862034 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":110,"threads_started":1}
I20260810 19:02:46.864339 27207 heartbeater.cc:347] Connected to a master server at 127.27.147.190:43411
I20260810 19:02:46.864568 27207 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:46.865039 27207 heartbeater.cc:511] Master 127.27.147.190:43411 requested a full tablet report, sending...
I20260810 19:02:46.866634 26908 ts_manager.cc:194] Registered new tserver with Master: 42ce2a0c7fd74a9a9de876c3d3465dbf (127.27.147.129:39567)
I20260810 19:02:46.867810 26908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.129:39507
W20260810 19:02:46.972496 27211 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:46.972774 27211 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:46.972855 27211 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:46.976806 27211 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:46.976886 27211 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:46.976974 27211 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.130
I20260810 19:02:46.981578 27211 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.27.147.130
--webserver_port=0
--tserver_master_addrs=127.27.147.190:43411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27211
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.130
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:46.982537 27211 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:46.983891 27211 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:46.993463 27224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:46.993481 27221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:46.993530 27211 server_base.cc:1034] running on GCE node
W20260810 19:02:46.993481 27222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:46.994143 27211 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:46.994720 27211 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:46.995883 27211 hybrid_clock.cc:648] HybridClock initialized: now 1786388566995865 us; error 30 us; skew 500 ppm
I20260810 19:02:46.997752 27211 webserver.cc:460] Webserver started at http://127.27.147.130:39049/ using document root <none> and password file <none>
I20260810 19:02:46.998274 27211 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:46.998345 27211 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:46.998507 27211 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:47.000473 27211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data/instance:
uuid: "ed05d6c7c2814c3aa0b86fee5878382b"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "16103c678301452c9637f9d0f2079dd7"
server_key_iv: "6559cb0f03f92d63e00c8815b5665e2c"
server_key_version: "encryptionkey@0"
I20260810 19:02:47.000965 27211 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance:
uuid: "ed05d6c7c2814c3aa0b86fee5878382b"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "16103c678301452c9637f9d0f2079dd7"
server_key_iv: "6559cb0f03f92d63e00c8815b5665e2c"
server_key_version: "encryptionkey@0"
I20260810 19:02:47.004514 27211 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 19:02:47.006775 27231 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:47.008131 27211 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:47.008234 27211 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal
uuid: "ed05d6c7c2814c3aa0b86fee5878382b"
format_stamp: "Formatted at 2026-08-10 19:02:46 on dist-test-slave-znh6"
server_key: "16103c678301452c9637f9d0f2079dd7"
server_key_iv: "6559cb0f03f92d63e00c8815b5665e2c"
server_key_version: "encryptionkey@0"
I20260810 19:02:47.008316 27211 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:47.023769 27211 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:47.024291 27211 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:47.024434 27211 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:47.024907 27211 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:47.025736 27211 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:47.025782 27211 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:47.025820 27211 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:47.025835 27211 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:47.034032 27211 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.130:43301
I20260810 19:02:47.034128 27390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.130:43301 every 8 connection(s)
I20260810 19:02:47.035009 27211 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/data/info.pb
I20260810 19:02:47.037735 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 27211
I20260810 19:02:47.037865 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal/instance
I20260810 19:02:47.038199 28238 external_mini_cluster.cc:1428] Setting key 3c3a164da92b6f06bc1dd3fad82db7fd
I20260810 19:02:47.040395 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.27.147.131:0
--local_ip_for_outbound_sockets=127.27.147.131
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:43411
--builtin_ntp_servers=127.27.147.148:36889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 19:02:47.045090 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.036746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.130:37079 (local address 127.27.147.190:43411)
0810 19:02:47.036885 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.036889 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.037878 (+ 989us) server_negotiation.cc:407] Connection header received
0810 19:02:47.039595 (+ 1717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.039600 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.039654 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.039732 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:47.040682 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.041503 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.042659 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.043111 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.043259 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.044476 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:47.044483 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:47.044489 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:47.044506 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:47.044535 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:47.044548 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:47.044625 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.044923 (+ 298us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.044980 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:02:47.046440 27393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.037112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:43411 (local address 127.27.147.130:37079)
0810 19:02:47.037726 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.037763 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:47.039369 (+ 1606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:47.039932 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:47.039939 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:47.040018 (+ 79us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:47.040520 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.040543 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.041658 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.041662 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.042534 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.042543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.043272 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.043275 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.043538 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.044233 (+ 695us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:47.044234 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:47.044272 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:47.044329 (+ 57us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:47.044675 (+ 346us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:47.044689 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:47.044698 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 19:02:47.044906 (+ 208us) client_negotiation.cc:240] Negotiation successful
0810 19:02:47.045327 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"thread_start_us":119,"threads_started":1}
I20260810 19:02:47.047679 27391 heartbeater.cc:347] Connected to a master server at 127.27.147.190:43411
I20260810 19:02:47.047904 27391 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:47.048386 27391 heartbeater.cc:511] Master 127.27.147.190:43411 requested a full tablet report, sending...
I20260810 19:02:47.049433 26908 ts_manager.cc:194] Registered new tserver with Master: ed05d6c7c2814c3aa0b86fee5878382b (127.27.147.130:43301)
I20260810 19:02:47.050001 26908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.130:37079
W20260810 19:02:47.153568 27394 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:47.153841 27394 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:47.153923 27394 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:47.157948 27394 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:47.158022 27394 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:47.158105 27394 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.27.147.131
I20260810 19:02:47.162665 27394 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.27.147.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.27.147.131
--webserver_port=0
--tserver_master_addrs=127.27.147.190:43411
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27394
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.27.147.131
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:47.163686 27394 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:47.165000 27394 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:47.174435 27405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:47.174448 27403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:47.174453 27408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:47.174657 27394 server_base.cc:1034] running on GCE node
I20260810 19:02:47.175138 27394 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:47.175748 27394 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:47.176945 27394 hybrid_clock.cc:648] HybridClock initialized: now 1786388567176915 us; error 41 us; skew 500 ppm
I20260810 19:02:47.178727 27394 webserver.cc:460] Webserver started at http://127.27.147.131:45815/ using document root <none> and password file <none>
I20260810 19:02:47.179318 27394 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:47.179416 27394 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:47.179653 27394 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:47.181563 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data/instance:
uuid: "7e8c3a32de46451684361bbba6479ddd"
format_stamp: "Formatted at 2026-08-10 19:02:47 on dist-test-slave-znh6"
server_key: "e2a0fbd1d8a37dfc05b9601fbb92d050"
server_key_iv: "e7bf6b2092284c1f8dfdcdb5adcd254f"
server_key_version: "encryptionkey@0"
I20260810 19:02:47.182114 27394 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance:
uuid: "7e8c3a32de46451684361bbba6479ddd"
format_stamp: "Formatted at 2026-08-10 19:02:47 on dist-test-slave-znh6"
server_key: "e2a0fbd1d8a37dfc05b9601fbb92d050"
server_key_iv: "e7bf6b2092284c1f8dfdcdb5adcd254f"
server_key_version: "encryptionkey@0"
I20260810 19:02:47.185649 27394 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 19:02:47.188604 27417 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:47.189762 27394 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 19:02:47.189883 27394 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal
uuid: "7e8c3a32de46451684361bbba6479ddd"
format_stamp: "Formatted at 2026-08-10 19:02:47 on dist-test-slave-znh6"
server_key: "e2a0fbd1d8a37dfc05b9601fbb92d050"
server_key_iv: "e7bf6b2092284c1f8dfdcdb5adcd254f"
server_key_version: "encryptionkey@0"
I20260810 19:02:47.189989 27394 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:47.213565 27394 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:47.214174 27394 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:47.214370 27394 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:47.214982 27394 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 19:02:47.215863 27394 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 19:02:47.215911 27394 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:47.215981 27394 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 19:02:47.216019 27394 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:47.224913 27394 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.131:40511
I20260810 19:02:47.224936 27584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.131:40511 every 8 connection(s)
I20260810 19:02:47.225818 27394 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/data/info.pb
I20260810 19:02:47.228212 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 27394
I20260810 19:02:47.228348 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal/instance
I20260810 19:02:47.228669 28238 external_mini_cluster.cc:1428] Setting key c88ad1fbf28957d62f934a3591b8fa7a
I20260810 19:02:47.235214 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.227655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.131:52829 (local address 127.27.147.190:43411)
0810 19:02:47.227822 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.227828 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.229002 (+ 1174us) server_negotiation.cc:407] Connection header received
0810 19:02:47.230726 (+ 1724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.230729 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.230813 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.230916 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:47.231869 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.232401 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.233476 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.233688 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.233766 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.234800 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:47.234803 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:47.234805 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:47.234816 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:47.234853 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:47.234861 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:47.234917 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.235047 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.235105 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 19:02:47.236063 27587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.228024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:43411 (local address 127.27.147.131:52829)
0810 19:02:47.228815 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.228870 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:47.230464 (+ 1594us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:47.231183 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:47.231190 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:47.231273 (+ 83us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:02:47.231736 (+ 463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.231748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.232518 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.232521 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.233364 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.233373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.233813 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.233815 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.233950 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.234570 (+ 620us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 19:02:47.234571 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:02:47.234609 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 19:02:47.234669 (+ 60us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:02:47.234920 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:02:47.234926 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:02:47.234930 (+ 4us) client_negotiation.cc:769] Sending connection context
0810 19:02:47.235030 (+ 100us) client_negotiation.cc:240] Negotiation successful
0810 19:02:47.235232 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"thread_start_us":112,"threads_started":1}
I20260810 19:02:47.237110 27585 heartbeater.cc:347] Connected to a master server at 127.27.147.190:43411
I20260810 19:02:47.237360 27585 heartbeater.cc:464] Registering TS with master...
I20260810 19:02:47.237861 27585 heartbeater.cc:511] Master 127.27.147.190:43411 requested a full tablet report, sending...
I20260810 19:02:47.238854 26908 ts_manager.cc:194] Registered new tserver with Master: 7e8c3a32de46451684361bbba6479ddd (127.27.147.131:40511)
I20260810 19:02:47.239434 26908 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.27.147.131:52829
I20260810 19:02:47.242661 28238 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 19:02:47.248435 26977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.244800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41346 (local address 127.27.147.190:43411)
0810 19:02:47.244908 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.244914 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.245193 (+ 279us) server_negotiation.cc:407] Connection header received
0810 19:02:47.245293 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.245297 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.245341 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.245415 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:47.245632 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.246179 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.247041 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.247239 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.247331 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.248061 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:47.248064 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:47.248065 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:47.248075 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:47.248089 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:47.248095 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:47.248147 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.248222 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.248276 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260810 19:02:47.250941 26908 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:41346:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 19:02:47.253872 26908 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 19:02:47.267745 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.261979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43301 (local address 127.0.0.1:36748)
0810 19:02:47.263016 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.263029 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:47.263182 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:47.263810 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:47.263812 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:47.263827 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:47.263963 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.263973 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.264989 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.264992 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.266210 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.266220 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.267200 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.267204 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.267316 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.267357 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 19:02:47.267480 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 19:02:47.267601 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":974,"mutex_wait_us":41,"thread_start_us":82,"threads_started":1}
I20260810 19:02:47.267809 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.262388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.27.147.130:43301)
0810 19:02:47.262863 (+ 475us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.262877 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.263106 (+ 229us) server_negotiation.cc:407] Connection header received
0810 19:02:47.263349 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.263353 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.263501 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.263631 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:47.264124 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.264863 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.266336 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.267070 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.267194 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.267356 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.267543 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.267644 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":351,"thread_start_us":134,"threads_started":1}
I20260810 19:02:47.268195 27601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.261816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:39567 (local address 127.0.0.1:39230)
0810 19:02:47.262850 (+ 1034us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.262906 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:47.263106 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:47.263651 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:47.263657 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:47.263678 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:47.263794 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.263801 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.265266 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.265272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.266620 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.266631 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.267795 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.267800 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.267890 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.267923 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 19:02:47.267993 (+ 70us) client_negotiation.cc:240] Negotiation successful
0810 19:02:47.268065 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":840,"thread_start_us":73,"threads_started":1}
I20260810 19:02:47.268652 27605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.262514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:40511 (local address 127.0.0.1:54304)
0810 19:02:47.263375 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.263387 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:47.263500 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:47.263944 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:47.263947 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:47.263964 (+ 17us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:47.264054 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.264061 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.265352 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.265355 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.266538 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.266549 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.268220 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.268226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.268357 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.268381 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 19:02:47.268456 (+ 75us) client_negotiation.cc:240] Negotiation successful
0810 19:02:47.268523 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"mutex_wait_us":3,"thread_start_us":98,"threads_started":1}
I20260810 19:02:47.268677 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.261966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39230 (local address 127.27.147.129:39567)
0810 19:02:47.262569 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.262591 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.262997 (+ 406us) server_negotiation.cc:407] Connection header received
0810 19:02:47.263285 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.263294 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.263502 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.263640 (+ 138us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:47.263920 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.265105 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.266722 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.267456 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.267571 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.267725 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.268416 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.268515 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":484,"thread_start_us":136,"threads_started":1}
I20260810 19:02:47.269145 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.262923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.27.147.131:40511)
0810 19:02:47.263321 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.263333 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.263475 (+ 142us) server_negotiation.cc:407] Connection header received
0810 19:02:47.263632 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.263637 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.263816 (+ 179us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.263958 (+ 142us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:47.264166 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.265242 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.266676 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.267473 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.267588 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.267740 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.268858 (+ 1118us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.268980 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":286,"thread_start_us":93,"threads_started":1}
I20260810 19:02:47.270946 27296 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd99e0bc3bd475ba16391dc80345dbd (DEFAULT_TABLE table=test-table [id=cace2d4842034f889811c4a171891853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:47.271239 27101 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd99e0bc3bd475ba16391dc80345dbd (DEFAULT_TABLE table=test-table [id=cace2d4842034f889811c4a171891853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:47.272071 27296 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd99e0bc3bd475ba16391dc80345dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:47.271706 27482 tablet_service.cc:1467] Processing CreateTablet for tablet 7fd99e0bc3bd475ba16391dc80345dbd (DEFAULT_TABLE table=test-table [id=cace2d4842034f889811c4a171891853]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 19:02:47.272159 27101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd99e0bc3bd475ba16391dc80345dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:47.272606 27482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7fd99e0bc3bd475ba16391dc80345dbd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:47.278807 27607 tablet_bootstrap.cc:492] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: Bootstrap starting.
I20260810 19:02:47.279284 27609 tablet_bootstrap.cc:492] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Bootstrap starting.
I20260810 19:02:47.279987 27608 tablet_bootstrap.cc:492] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Bootstrap starting.
I20260810 19:02:47.281209 27607 tablet_bootstrap.cc:654] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:47.281209 27609 tablet_bootstrap.cc:654] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:47.281678 27608 tablet_bootstrap.cc:654] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:47.282092 27609 log.cc:824] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:47.282092 27607 log.cc:824] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:47.282398 27608 log.cc:824] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:47.284193 27609 tablet_bootstrap.cc:492] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: No bootstrap required, opened a new log
I20260810 19:02:47.284195 27607 tablet_bootstrap.cc:492] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: No bootstrap required, opened a new log
I20260810 19:02:47.284356 27608 tablet_bootstrap.cc:492] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: No bootstrap required, opened a new log
I20260810 19:02:47.284389 27609 ts_tablet_manager.cc:1397] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 19:02:47.284403 27607 ts_tablet_manager.cc:1397] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:02:47.284572 27608 ts_tablet_manager.cc:1397] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260810 19:02:47.287736 27608 raft_consensus.cc:348] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.287971 27608 raft_consensus.cc:374] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:47.288036 27608 raft_consensus.cc:729] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed05d6c7c2814c3aa0b86fee5878382b, State: Initialized, Role: FOLLOWER
I20260810 19:02:47.288475 27608 consensus_queue.cc:260] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.288637 27609 raft_consensus.cc:348] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.288637 27607 raft_consensus.cc:348] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.288878 27607 raft_consensus.cc:374] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:47.288878 27609 raft_consensus.cc:374] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:47.288944 27609 raft_consensus.cc:729] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e8c3a32de46451684361bbba6479ddd, State: Initialized, Role: FOLLOWER
I20260810 19:02:47.288944 27607 raft_consensus.cc:729] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42ce2a0c7fd74a9a9de876c3d3465dbf, State: Initialized, Role: FOLLOWER
I20260810 19:02:47.289254 27391 heartbeater.cc:503] Master 127.27.147.190:43411 was elected leader, sending a full tablet report...
I20260810 19:02:47.289363 27609 consensus_queue.cc:260] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.289363 27607 consensus_queue.cc:260] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.289533 27608 ts_tablet_manager.cc:1428] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20260810 19:02:47.290273 27607 ts_tablet_manager.cc:1428] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260810 19:02:47.290273 27609 ts_tablet_manager.cc:1428] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260810 19:02:47.290359 27585 heartbeater.cc:503] Master 127.27.147.190:43411 was elected leader, sending a full tablet report...
I20260810 19:02:47.291561 27207 heartbeater.cc:503] Master 127.27.147.190:43411 was elected leader, sending a full tablet report...
W20260810 19:02:47.353726 27208 tablet.cc:2369] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:02:47.355834 27619 raft_consensus.cc:482] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 19:02:47.356038 27619 raft_consensus.cc:504] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.357354 27619 leader_election.cc:288] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ed05d6c7c2814c3aa0b86fee5878382b (127.27.147.130:43301), 7e8c3a32de46451684361bbba6479ddd (127.27.147.131:40511)
I20260810 19:02:47.362203 27603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.357835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:48321 (local address 127.27.147.130:43301)
0810 19:02:47.357981 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.357986 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.358007 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:02:47.358054 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.358057 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.358096 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.358198 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:47.358529 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.359366 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.360952 (+ 1586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.361554 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.361680 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.361731 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.361998 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.362072 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:02:47.362205 27606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.357708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.27.147.129:52641 (local address 127.27.147.131:40511)
0810 19:02:47.357843 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.357847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.358156 (+ 309us) server_negotiation.cc:407] Connection header received
0810 19:02:47.358263 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.358266 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.358314 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.358389 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 19:02:47.358681 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.359445 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.360952 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.361565 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.361680 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.361731 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.361998 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.362072 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 19:02:47.362214 27210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.357608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.130:43301 (local address 127.27.147.129:48321)
0810 19:02:47.357760 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.357777 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:47.357876 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:47.358234 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:47.358237 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:47.358256 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:47.358391 (+ 135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.358400 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.359478 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.359481 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.360817 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.360829 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.361695 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.361700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.361834 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.361873 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 19:02:47.361963 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 19:02:47.362057 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53,"mutex_wait_us":73}
I20260810 19:02:47.362223 27620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.357608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.131:40511 (local address 127.27.147.129:52641)
0810 19:02:47.358055 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.358067 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:02:47.358151 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:02:47.358425 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:02:47.358430 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:02:47.358453 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 19:02:47.358571 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.358576 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.359539 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.359542 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.360821 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:02:47.360829 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.361695 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.361700 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:02:47.361828 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.361880 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 19:02:47.361959 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 19:02:47.362057 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":382,"mutex_wait_us":204,"thread_start_us":104,"threads_started":1}
I20260810 19:02:47.363008 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd99e0bc3bd475ba16391dc80345dbd" candidate_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" is_pre_election: true
I20260810 19:02:47.363206 27511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd99e0bc3bd475ba16391dc80345dbd" candidate_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8c3a32de46451684361bbba6479ddd" is_pre_election: true
I20260810 19:02:47.363333 27325 raft_consensus.cc:2454] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ce2a0c7fd74a9a9de876c3d3465dbf in term 0.
I20260810 19:02:47.363539 27511 raft_consensus.cc:2454] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42ce2a0c7fd74a9a9de876c3d3465dbf in term 0.
I20260810 19:02:47.363873 27029 leader_election.cc:302] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ce2a0c7fd74a9a9de876c3d3465dbf, ed05d6c7c2814c3aa0b86fee5878382b; no voters:
I20260810 19:02:47.364182 27619 raft_consensus.cc:2781] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 19:02:47.364270 27619 raft_consensus.cc:482] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 19:02:47.364313 27619 raft_consensus.cc:3037] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:47.365319 27619 raft_consensus.cc:504] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.365670 27619 leader_election.cc:288] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [CANDIDATE]: Term 1 election: Requested vote from peers ed05d6c7c2814c3aa0b86fee5878382b (127.27.147.130:43301), 7e8c3a32de46451684361bbba6479ddd (127.27.147.131:40511)
I20260810 19:02:47.366005 27325 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd99e0bc3bd475ba16391dc80345dbd" candidate_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ed05d6c7c2814c3aa0b86fee5878382b"
I20260810 19:02:47.366055 27511 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7fd99e0bc3bd475ba16391dc80345dbd" candidate_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e8c3a32de46451684361bbba6479ddd"
I20260810 19:02:47.366137 27325 raft_consensus.cc:3037] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:47.366155 27511 raft_consensus.cc:3037] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:47.367445 27511 raft_consensus.cc:2454] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ce2a0c7fd74a9a9de876c3d3465dbf in term 1.
I20260810 19:02:47.367445 27325 raft_consensus.cc:2454] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42ce2a0c7fd74a9a9de876c3d3465dbf in term 1.
I20260810 19:02:47.367776 27033 leader_election.cc:302] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42ce2a0c7fd74a9a9de876c3d3465dbf, 7e8c3a32de46451684361bbba6479ddd; no voters:
I20260810 19:02:47.367961 27619 raft_consensus.cc:2781] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:47.368181 27619 raft_consensus.cc:686] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [term 1 LEADER]: Becoming Leader. State: Replica: 42ce2a0c7fd74a9a9de876c3d3465dbf, State: Running, Role: LEADER
I20260810 19:02:47.368508 27619 consensus_queue.cc:237] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } }
I20260810 19:02:47.371680 26907 catalog_manager.cc:5495] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf reported cstate change: term changed from 0 to 1, leader changed from <none> to 42ce2a0c7fd74a9a9de876c3d3465dbf (127.27.147.129). New cstate: current_term: 1 leader_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42ce2a0c7fd74a9a9de876c3d3465dbf" member_type: VOTER last_known_addr { host: "127.27.147.129" port: 39567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 } health_report { overall_health: UNKNOWN } } }
I20260810 19:02:47.388527 27602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.384838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39236 (local address 127.27.147.129:39567)
0810 19:02:47.384975 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:47.384979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:47.384993 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:02:47.385147 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:47.385150 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:47.385210 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:47.385312 (+ 102us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:02:47.385564 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.386057 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.387120 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:47.387341 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:47.387457 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:47.387757 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:02:47.388120 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:02:47.388249 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:47.388311 (+ 62us) server_negotiation.cc:299] Negotiation successful
0810 19:02:47.388359 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:02:47.393788 27325 raft_consensus.cc:1261] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 1 FOLLOWER]: Refusing update from remote peer 42ce2a0c7fd74a9a9de876c3d3465dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:47.393788 27511 raft_consensus.cc:1261] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 1 FOLLOWER]: Refusing update from remote peer 42ce2a0c7fd74a9a9de876c3d3465dbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:02:47.394474 27621 consensus_queue.cc:1035] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e8c3a32de46451684361bbba6479ddd" member_type: VOTER last_known_addr { host: "127.27.147.131" port: 40511 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:47.394630 27619 consensus_queue.cc:1035] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed05d6c7c2814c3aa0b86fee5878382b" member_type: VOTER last_known_addr { host: "127.27.147.130" port: 43301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:02:47.401309 27624 mvcc.cc:205] Tried to move back new op lower bound from 7317047572038377472 to 7317047571944050688. Current Snapshot: MvccSnapshot[applied={T|T < 7317047572038377472}]
I20260810 19:02:47.401803 27625 mvcc.cc:205] Tried to move back new op lower bound from 7317047572038377472 to 7317047571944050688. Current Snapshot: MvccSnapshot[applied={T|T < 7317047572038377472}]
I20260810 19:02:47.404351 27626 mvcc.cc:205] Tried to move back new op lower bound from 7317047572038377472 to 7317047571944050688. Current Snapshot: MvccSnapshot[applied={T|T < 7317047572038377472}]
I20260810 19:02:47.410297 26906 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:41346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:47.410477 26906 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:41346:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 19:02:47.413028 26906 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0361b68f96584723a54480c115969e74: Sending DeleteTablet for 3 replicas of tablet 7fd99e0bc3bd475ba16391dc80345dbd
I20260810 19:02:47.413770 27296 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd99e0bc3bd475ba16391dc80345dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:47 UTC) from {username='slave'} at 127.0.0.1:36748
I20260810 19:02:47.413807 27101 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd99e0bc3bd475ba16391dc80345dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:47 UTC) from {username='slave'} at 127.0.0.1:39230
I20260810 19:02:47.413923 27482 tablet_service.cc:1514] Processing DeleteTablet for tablet 7fd99e0bc3bd475ba16391dc80345dbd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 19:02:47 UTC) from {username='slave'} at 127.0.0.1:54304
I20260810 19:02:47.414335 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 27010
I20260810 19:02:47.414356 27641 tablet_replica.cc:330] T 7fd99e0bc3bd475ba16391dc80345dbd P 42ce2a0c7fd74a9a9de876c3d3465dbf: stopping tablet replica
I20260810 19:02:47.420449 27642 tablet_replica.cc:330] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: stopping tablet replica
I20260810 19:02:47.420753 27642 raft_consensus.cc:2229] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:47.421056 27642 raft_consensus.cc:2258] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:47.421470 27639 tablet_replica.cc:330] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: stopping tablet replica
I20260810 19:02:47.421733 27639 raft_consensus.cc:2229] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 19:02:47.422005 27639 raft_consensus.cc:2258] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 19:02:47.422670 27642 ts_tablet_manager.cc:1905] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:47.423291 27639 ts_tablet_manager.cc:1905] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 19:02:47.425405 27639 ts_tablet_manager.cc:1918] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:47.425436 27642 ts_tablet_manager.cc:1918] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 19:02:47.425520 27642 log.cc:1196] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-2/wal/wals/7fd99e0bc3bd475ba16391dc80345dbd
I20260810 19:02:47.425521 27639 log.cc:1196] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Deleting WAL directory at /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786388305279399-28238-0/minicluster-data/ts-1/wal/wals/7fd99e0bc3bd475ba16391dc80345dbd
I20260810 19:02:47.425814 27639 ts_tablet_manager.cc:1939] T 7fd99e0bc3bd475ba16391dc80345dbd P ed05d6c7c2814c3aa0b86fee5878382b: Deleting consensus metadata
I20260810 19:02:47.425822 27642 ts_tablet_manager.cc:1939] T 7fd99e0bc3bd475ba16391dc80345dbd P 7e8c3a32de46451684361bbba6479ddd: Deleting consensus metadata
I20260810 19:02:47.426946 26887 catalog_manager.cc:4841] TS ed05d6c7c2814c3aa0b86fee5878382b (127.27.147.130:43301): tablet 7fd99e0bc3bd475ba16391dc80345dbd (table test-table [id=cace2d4842034f889811c4a171891853]) successfully deleted
I20260810 19:02:47.427034 26893 catalog_manager.cc:4841] TS 7e8c3a32de46451684361bbba6479ddd (127.27.147.131:40511): tablet 7fd99e0bc3bd475ba16391dc80345dbd (table test-table [id=cace2d4842034f889811c4a171891853]) successfully deleted
W20260810 19:02:47.427660 26893 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.27.147.129:39567 (error 108)
I20260810 19:02:47.428407 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 27211
I20260810 19:02:47.428683 27600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:47.428339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.129:39567 (local address 127.0.0.1:39248)
0810 19:02:47.428502 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 19:02:47.428584 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.129:39567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 19:02:47.434945 26893 catalog_manager.cc:4584] TS 42ce2a0c7fd74a9a9de876c3d3465dbf (127.27.147.129:39567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fd99e0bc3bd475ba16391dc80345dbd: Network error: Client connection negotiation failed: client connection to 127.27.147.129:39567: connect: Connection refused (error 111)
I20260810 19:02:47.441702 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 27394
I20260810 19:02:47.457942 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 26868
2026-08-10T19:02:47Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1034 ms)
[----------] 6 tests from EncryptionPolicyTest (11100 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:47 dist-test-slave-znh6 krb5kdc[27647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:47 dist-test-slave-znh6 krb5kdc[27647](info): set up 2 sockets
Aug 10 19:02:47 dist-test-slave-znh6 krb5kdc[27647](info): commencing operation
krb5kdc: starting...
W20260810 19:02:48.702997 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.217s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:48 dist-test-slave-znh6 krb5kdc[27647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:02:48.829699 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40153
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34209
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:48.942056 27663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:48.942344 27663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:48.942416 27663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:48.946363 27663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:48.946452 27663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:48.946492 27663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:48.946547 27663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:48.946583 27663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:48.951843 27663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34209
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40153
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27663
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:48.952847 27663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:48.953845 27663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:48.962590 27672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:48.962630 27670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:48.962636 27663 server_base.cc:1034] running on GCE node
W20260810 19:02:48.962590 27669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:48.963469 27663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:48.964556 27663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:48.965728 27663 hybrid_clock.cc:648] HybridClock initialized: now 1786388568965714 us; error 25 us; skew 500 ppm
Aug 10 19:02:48 dist-test-slave-znh6 krb5kdc[27647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388568, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:48.968467 27663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:02:48.969367 27663 webserver.cc:460] Webserver started at http://127.0.0.1:35599/ using document root <none> and password file <none>
I20260810 19:02:48.969923 27663 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:48.970016 27663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:48.970249 27663 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:48.972289 27663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "33be79f66fbd4eddbcd281bc36edf234"
format_stamp: "Formatted at 2026-08-10 19:02:48 on dist-test-slave-znh6"
server_key: "0bca749eecbbcf73030c3c315698f054"
server_key_iv: "e2464d4f163672b18ca11e2cd1aa7048"
server_key_version: "encryptionkey@0"
I20260810 19:02:48.972849 27663 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "33be79f66fbd4eddbcd281bc36edf234"
format_stamp: "Formatted at 2026-08-10 19:02:48 on dist-test-slave-znh6"
server_key: "0bca749eecbbcf73030c3c315698f054"
server_key_iv: "e2464d4f163672b18ca11e2cd1aa7048"
server_key_version: "encryptionkey@0"
I20260810 19:02:48.976480 27663 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:02:48.978971 27685 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:48.980252 27663 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:02:48.980425 27663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "33be79f66fbd4eddbcd281bc36edf234"
format_stamp: "Formatted at 2026-08-10 19:02:48 on dist-test-slave-znh6"
server_key: "0bca749eecbbcf73030c3c315698f054"
server_key_iv: "e2464d4f163672b18ca11e2cd1aa7048"
server_key_version: "encryptionkey@0"
I20260810 19:02:48.980541 27663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:49.012832 27663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:49.015740 27663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:49.015959 27663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:49.023237 27663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40153
I20260810 19:02:49.023283 27760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40153 every 8 connection(s)
I20260810 19:02:49.024183 27663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:49.026003 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 27663
I20260810 19:02:49.026140 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:49.026513 28238 external_mini_cluster.cc:1428] Setting key 21e05eb4c691e5592926161b7cb2da7e
I20260810 19:02:49.027464 27761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:49.034142 27761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234: Bootstrap starting.
Aug 10 19:02:49 dist-test-slave-znh6 krb5kdc[27647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388568, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:02:49.036600 27761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:49.037611 27761 log.cc:824] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:49.040020 27761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234: No bootstrap required, opened a new log
I20260810 19:02:49.041285 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:49.027921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48477 (local address 127.0.0.1:40153)
0810 19:02:49.028327 (+ 406us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:49.028346 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:49.028389 (+ 43us) server_negotiation.cc:407] Connection header received
0810 19:02:49.029020 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:49.029036 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:49.029351 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:49.029618 (+ 267us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:49.030635 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:49.031684 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:49.032884 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:49.033331 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:49.033443 (+ 112us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:49.036233 (+ 2790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:49.036238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:49.036250 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:49.036277 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:49.038212 (+ 1935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:49.038876 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:49.038882 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:49.038888 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:49.039024 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:49.039348 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:49.039352 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:49.039353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:49.039835 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:49.040044 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:49.040397 (+ 353us) server_negotiation.cc:299] Negotiation successful
0810 19:02:49.040608 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":139,"threads_started":1}
I20260810 19:02:49.044148 27761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be79f66fbd4eddbcd281bc36edf234" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260810 19:02:49.044414 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:02:49.044428 27761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:49.044512 27761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33be79f66fbd4eddbcd281bc36edf234, State: Initialized, Role: FOLLOWER
I20260810 19:02:49.044919 27761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be79f66fbd4eddbcd281bc36edf234" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260810 19:02:49.045037 27761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:49.045120 27761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:49.045225 27761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:49.046195 27761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be79f66fbd4eddbcd281bc36edf234" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260810 19:02:49.046574 27761 leader_election.cc:302] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33be79f66fbd4eddbcd281bc36edf234; no voters:
I20260810 19:02:49.046958 27761 leader_election.cc:288] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:49.047025 27780 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:49.047263 27780 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [term 1 LEADER]: Becoming Leader. State: Replica: 33be79f66fbd4eddbcd281bc36edf234, State: Running, Role: LEADER
I20260810 19:02:49.047591 27780 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be79f66fbd4eddbcd281bc36edf234" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } }
I20260810 19:02:49.048250 27761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:49.048738 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33be79f66fbd4eddbcd281bc36edf234. Latest consensus state: current_term: 1 leader_uuid: "33be79f66fbd4eddbcd281bc36edf234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be79f66fbd4eddbcd281bc36edf234" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } } }
I20260810 19:02:49.048929 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:49.049198 27780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33be79f66fbd4eddbcd281bc36edf234" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33be79f66fbd4eddbcd281bc36edf234" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40153 } } }
I20260810 19:02:49.049351 27780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:49.049672 27793 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:49.053220 27793 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:49.055704 27766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:49.046637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46727 (local address 127.0.0.1:40153)
0810 19:02:49.046753 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:49.046758 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:49.046999 (+ 241us) server_negotiation.cc:407] Connection header received
0810 19:02:49.047142 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:49.047144 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:49.047192 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:49.047279 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:49.047994 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:49.048754 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:49.049912 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:49.050256 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:49.050767 (+ 511us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:49.052021 (+ 1254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:49.052026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:49.052028 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:49.052053 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:49.053640 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:49.054239 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:49.054242 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:49.054244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:49.054323 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:49.054696 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:49.054699 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:49.054701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:49.054955 (+ 254us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:49.055094 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:49.055437 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 19:02:49.055523 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:02:49.058840 27793 catalog_manager.cc:1347] Generated new cluster ID: 0eade7b9c08c4732b8278be3ec9c6c54
I20260810 19:02:49.058916 27793 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:49.072857 27793 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:49.073660 27793 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:49.093977 27793 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33be79f66fbd4eddbcd281bc36edf234: Generated new TSK 0
I20260810 19:02:49.094625 27793 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:49.098603 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 27663
W20260810 19:02:49.117071 27645 connection.cc:535] client connection to 127.0.0.1:40153 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40153): Connection reset by peer (error 104)
2026-08-10T19:02:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1645 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:49 dist-test-slave-znh6 krb5kdc[27814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:49 dist-test-slave-znh6 krb5kdc[27814](info): set up 2 sockets
Aug 10 19:02:49 dist-test-slave-znh6 krb5kdc[27814](info): commencing operation
krb5kdc: starting...
W20260810 19:02:50.350154 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.219s user 0.004s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:50 dist-test-slave-znh6 krb5kdc[27814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388570, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:02:50.477376 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43987
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43987
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:50.588704 27833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:50.588960 27833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:50.589040 27833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:50.593274 27833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:50.593334 27833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:50.593354 27833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:50.593371 27833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:50.593415 27833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:50.598652 27833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43987
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:50.599700 27833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:50.600699 27833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:50.609565 27840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:50.609612 27833 server_base.cc:1034] running on GCE node
W20260810 19:02:50.609678 27839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:50.609602 27844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:50.610318 27833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:50.611423 27833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:50.612622 27833 hybrid_clock.cc:648] HybridClock initialized: now 1786388570612591 us; error 42 us; skew 500 ppm
Aug 10 19:02:50 dist-test-slave-znh6 krb5kdc[27814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388570, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:50.615391 27833 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:02:50.616297 27833 webserver.cc:460] Webserver started at http://127.0.0.1:43413/ using document root <none> and password file <none>
I20260810 19:02:50.616860 27833 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:50.616958 27833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:50.617194 27833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:50.619478 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "a3bb053819d340de9f0fbddcff55c903"
format_stamp: "Formatted at 2026-08-10 19:02:50 on dist-test-slave-znh6"
server_key: "d0f1216e679b0c3934da61517e225b21"
server_key_iv: "ba494d76c48760e67aacc49a02c14b4a"
server_key_version: "encryptionkey@0"
I20260810 19:02:50.620095 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "a3bb053819d340de9f0fbddcff55c903"
format_stamp: "Formatted at 2026-08-10 19:02:50 on dist-test-slave-znh6"
server_key: "d0f1216e679b0c3934da61517e225b21"
server_key_iv: "ba494d76c48760e67aacc49a02c14b4a"
server_key_version: "encryptionkey@0"
I20260810 19:02:50.623793 27833 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:02:50.626209 27860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:50.627431 27833 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:02:50.627584 27833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "a3bb053819d340de9f0fbddcff55c903"
format_stamp: "Formatted at 2026-08-10 19:02:50 on dist-test-slave-znh6"
server_key: "d0f1216e679b0c3934da61517e225b21"
server_key_iv: "ba494d76c48760e67aacc49a02c14b4a"
server_key_version: "encryptionkey@0"
I20260810 19:02:50.627694 27833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:50.650301 27833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:50.653333 27833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:50.653544 27833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:50.660908 27833 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43987
I20260810 19:02:50.660938 27941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43987 every 8 connection(s)
I20260810 19:02:50.661839 27833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:50.663403 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 27833
I20260810 19:02:50.663542 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:50.663887 28238 external_mini_cluster.cc:1428] Setting key fadb0b444db126131ef04b7b5408710b
I20260810 19:02:50.664999 27942 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:50.670570 27942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903: Bootstrap starting.
I20260810 19:02:50.672534 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:50.665288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41003 (local address 127.0.0.1:43987)
0810 19:02:50.665726 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:50.665740 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:50.665776 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:02:50.666399 (+ 623us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:50.666415 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:50.666727 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:50.667117 (+ 390us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:50.667507 (+ 390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:50.668474 (+ 967us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:50.669392 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:50.669847 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:50.670021 (+ 174us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:50.670959 (+ 938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:50.670964 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:50.670976 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:50.671332 (+ 356us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:50.671393 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:50.671416 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:50.671561 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:50.671744 (+ 183us) server_negotiation.cc:299] Negotiation successful
0810 19:02:50.671874 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":147,"threads_started":1}
I20260810 19:02:50.672881 27942 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:50.673684 27942 log.cc:824] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:50.675500 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:02:50.675850 27942 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903: No bootstrap required, opened a new log
I20260810 19:02:50.680035 27942 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bb053819d340de9f0fbddcff55c903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260810 19:02:50.680253 27942 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:50.680322 27942 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3bb053819d340de9f0fbddcff55c903, State: Initialized, Role: FOLLOWER
I20260810 19:02:50.680738 27942 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bb053819d340de9f0fbddcff55c903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260810 19:02:50.680902 27942 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:50.680976 27942 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:50.681092 27942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:50.681838 27947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:50.677466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56123 (local address 127.0.0.1:43987)
0810 19:02:50.677595 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:50.677599 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:50.677832 (+ 233us) server_negotiation.cc:407] Connection header received
0810 19:02:50.677975 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:50.677977 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:50.678011 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:50.678072 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:50.678383 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:50.678921 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:50.680016 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:50.680289 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:50.680421 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:50.681374 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:50.681378 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:50.681381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:50.681400 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:50.681430 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:50.681440 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:50.681538 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:50.681661 (+ 123us) server_negotiation.cc:299] Negotiation successful
0810 19:02:50.681697 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:02:50.681989 27942 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bb053819d340de9f0fbddcff55c903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260810 19:02:50.682331 27942 leader_election.cc:302] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3bb053819d340de9f0fbddcff55c903; no voters:
I20260810 19:02:50.682663 27942 leader_election.cc:288] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:50.682857 27954 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:50.683161 27954 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [term 1 LEADER]: Becoming Leader. State: Replica: a3bb053819d340de9f0fbddcff55c903, State: Running, Role: LEADER
I20260810 19:02:50.683523 27954 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bb053819d340de9f0fbddcff55c903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } }
I20260810 19:02:50.683912 27942 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:50.685192 27960 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3bb053819d340de9f0fbddcff55c903. Latest consensus state: current_term: 1 leader_uuid: "a3bb053819d340de9f0fbddcff55c903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bb053819d340de9f0fbddcff55c903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } } }
I20260810 19:02:50.685343 27960 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:50.685739 27957 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3bb053819d340de9f0fbddcff55c903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3bb053819d340de9f0fbddcff55c903" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43987 } } }
I20260810 19:02:50.685863 27957 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903 [sys.catalog]: This master's current role is: LEADER
W20260810 19:02:50.688686 27983 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:02:50.688781 27983 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:02:50.688871 27979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:50.689666 27979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:50.694784 27979 catalog_manager.cc:1347] Generated new cluster ID: 977356aa2b5148e3b4d60b0761de3a7e
I20260810 19:02:50.694869 27979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:50.713352 27979 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:50.714279 27979 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:50.717989 27979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3bb053819d340de9f0fbddcff55c903: Generated new TSK 0
I20260810 19:02:50.718714 27979 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:50.728513 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 27833
W20260810 19:02:50.749193 27810 connection.cc:535] client connection to 127.0.0.1:43987 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43987): Connection reset by peer (error 104)
2026-08-10T19:02:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1632 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:50 dist-test-slave-znh6 krb5kdc[27992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:50 dist-test-slave-znh6 krb5kdc[27992](info): set up 2 sockets
Aug 10 19:02:50 dist-test-slave-znh6 krb5kdc[27992](info): commencing operation
krb5kdc: starting...
W20260810 19:02:52.001206 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.238s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:52 dist-test-slave-znh6 krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388572, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:02:52.125908 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38831
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35217
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:52.236933 28010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:52.237259 28010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:52.237351 28010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:52.241400 28010 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:52.241489 28010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:52.241529 28010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:52.241587 28010 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:52.241626 28010 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:52.247004 28010 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38831
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35217
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35217
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:52.248103 28010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:52.249142 28010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:52.257762 28020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:52.257763 28018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:52.257781 28016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:52.258144 28010 server_base.cc:1034] running on GCE node
I20260810 19:02:52.258682 28010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:52.259804 28010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:52.261006 28010 hybrid_clock.cc:648] HybridClock initialized: now 1786388572260992 us; error 37 us; skew 500 ppm
Aug 10 19:02:52 dist-test-slave-znh6 krb5kdc[27992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388572, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:52.266804 28010 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:02:52.267741 28010 webserver.cc:460] Webserver started at http://127.0.0.1:33947/ using document root <none> and password file <none>
I20260810 19:02:52.268316 28010 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:52.268414 28010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:52.268656 28010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:52.270691 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "b1c3df135b7046689300572cd71ef0ff"
format_stamp: "Formatted at 2026-08-10 19:02:52 on dist-test-slave-znh6"
server_key: "fd3f43fd5e2b0e0d31c796d67f9c35fd"
server_key_iv: "5f1044963b91bed7efeae46a09555c7a"
server_key_version: "encryptionkey@0"
I20260810 19:02:52.271322 28010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "b1c3df135b7046689300572cd71ef0ff"
format_stamp: "Formatted at 2026-08-10 19:02:52 on dist-test-slave-znh6"
server_key: "fd3f43fd5e2b0e0d31c796d67f9c35fd"
server_key_iv: "5f1044963b91bed7efeae46a09555c7a"
server_key_version: "encryptionkey@0"
I20260810 19:02:52.275084 28010 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:02:52.277442 28033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:52.278596 28010 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:52.278738 28010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "b1c3df135b7046689300572cd71ef0ff"
format_stamp: "Formatted at 2026-08-10 19:02:52 on dist-test-slave-znh6"
server_key: "fd3f43fd5e2b0e0d31c796d67f9c35fd"
server_key_iv: "5f1044963b91bed7efeae46a09555c7a"
server_key_version: "encryptionkey@0"
I20260810 19:02:52.278864 28010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:52.283236 28010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:52.286085 28010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:52.286271 28010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:52.293407 28010 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35217
I20260810 19:02:52.293439 28119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35217 every 8 connection(s)
I20260810 19:02:52.294323 28010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:52.297076 28124 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:52.302111 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28010
I20260810 19:02:52.302259 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:52.302568 28238 external_mini_cluster.cc:1428] Setting key d71569d7740124271bedbcfc55b61fd7
I20260810 19:02:52.302644 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff: Bootstrap starting.
I20260810 19:02:52.305161 28124 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:52.305886 28124 log.cc:824] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:52.307579 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:52.303966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40787 (local address 127.0.0.1:35217)
0810 19:02:52.304421 (+ 455us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:52.304435 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:52.304471 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:02:52.305074 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:52.305090 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:52.305410 (+ 320us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:52.305734 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:52.306438 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:52.306443 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:52.306457 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:52.306474 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:52.306519 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:52.306538 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:52.306654 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:52.306753 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 19:02:52.306924 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":134,"threads_started":1}
I20260810 19:02:52.307983 28124 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff: No bootstrap required, opened a new log
I20260810 19:02:52.310420 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:02:52.312139 28124 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c3df135b7046689300572cd71ef0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260810 19:02:52.312379 28124 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:52.312443 28124 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1c3df135b7046689300572cd71ef0ff, State: Initialized, Role: FOLLOWER
I20260810 19:02:52.312925 28124 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c3df135b7046689300572cd71ef0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260810 19:02:52.313104 28124 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:52.313201 28124 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:52.313315 28124 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:52.314009 28129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:52.312465 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59415 (local address 127.0.0.1:35217)
0810 19:02:52.312623 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:52.312627 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:52.312765 (+ 138us) server_negotiation.cc:407] Connection header received
0810 19:02:52.312867 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:52.312870 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:52.312901 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:52.312979 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:52.313651 (+ 672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:52.313655 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:52.313658 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:52.313670 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:52.313696 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:52.313707 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:52.313809 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:52.313867 (+ 58us) server_negotiation.cc:299] Negotiation successful
0810 19:02:52.313900 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 19:02:52.314503 28124 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c3df135b7046689300572cd71ef0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260810 19:02:52.314869 28124 leader_election.cc:302] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1c3df135b7046689300572cd71ef0ff; no voters:
I20260810 19:02:52.315225 28124 leader_election.cc:288] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:52.315310 28142 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:52.315573 28142 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [term 1 LEADER]: Becoming Leader. State: Replica: b1c3df135b7046689300572cd71ef0ff, State: Running, Role: LEADER
I20260810 19:02:52.315912 28142 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c3df135b7046689300572cd71ef0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } }
I20260810 19:02:52.316426 28124 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:52.317811 28146 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1c3df135b7046689300572cd71ef0ff. Latest consensus state: current_term: 1 leader_uuid: "b1c3df135b7046689300572cd71ef0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c3df135b7046689300572cd71ef0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260810 19:02:52.317823 28145 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1c3df135b7046689300572cd71ef0ff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1c3df135b7046689300572cd71ef0ff" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35217 } } }
I20260810 19:02:52.317939 28146 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:52.317945 28145 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:52.318339 28165 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:52.321105 28165 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:52.326066 28165 catalog_manager.cc:1347] Generated new cluster ID: a8e70c27f7364264abd27a71699a22cc
I20260810 19:02:52.326130 28165 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:52.347256 28165 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:52.347630 28165 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:52.353895 28165 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1c3df135b7046689300572cd71ef0ff: Generated new TSK 0
I20260810 19:02:52.354487 28165 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:52.358369 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28010
2026-08-10T19:02:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1627 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:52 dist-test-slave-znh6 krb5kdc[28187](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:52 dist-test-slave-znh6 krb5kdc[28187](info): set up 2 sockets
Aug 10 19:02:52 dist-test-slave-znh6 krb5kdc[28187](info): commencing operation
krb5kdc: starting...
W20260810 19:02:53.615782 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:53 dist-test-slave-znh6 krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:02:53.741487 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36169
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36169
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:53.853507 28210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:53.853780 28210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:53.853853 28210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:53.857824 28210 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:53.857913 28210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:53.857951 28210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:53.858011 28210 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:53.858052 28210 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:53.863220 28210 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36169
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36169
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:53.864274 28210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:53.865278 28210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:53.873968 28218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:53.874043 28210 server_base.cc:1034] running on GCE node
W20260810 19:02:53.873991 28217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:53.873972 28222 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:53.874882 28210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:53.875955 28210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:53.877126 28210 hybrid_clock.cc:648] HybridClock initialized: now 1786388573877096 us; error 40 us; skew 500 ppm
Aug 10 19:02:53 dist-test-slave-znh6 krb5kdc[28187](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388573, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:53.879876 28210 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:02:53.880759 28210 webserver.cc:460] Webserver started at http://127.0.0.1:38077/ using document root <none> and password file <none>
I20260810 19:02:53.881264 28210 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:53.881325 28210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:53.881495 28210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:53.883546 28210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "5a5d8ce320de4fbe89f73da5d4e1b112"
format_stamp: "Formatted at 2026-08-10 19:02:53 on dist-test-slave-znh6"
server_key: "de7eadeee3743c75a27d74e4f2cb91ad"
server_key_iv: "df0b980a10e16a08e071fcc9852b0409"
server_key_version: "encryptionkey@0"
I20260810 19:02:53.884045 28210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "5a5d8ce320de4fbe89f73da5d4e1b112"
format_stamp: "Formatted at 2026-08-10 19:02:53 on dist-test-slave-znh6"
server_key: "de7eadeee3743c75a27d74e4f2cb91ad"
server_key_iv: "df0b980a10e16a08e071fcc9852b0409"
server_key_version: "encryptionkey@0"
I20260810 19:02:53.887652 28210 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:02:53.889887 28235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:53.891120 28210 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:02:53.891256 28210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "5a5d8ce320de4fbe89f73da5d4e1b112"
format_stamp: "Formatted at 2026-08-10 19:02:53 on dist-test-slave-znh6"
server_key: "de7eadeee3743c75a27d74e4f2cb91ad"
server_key_iv: "df0b980a10e16a08e071fcc9852b0409"
server_key_version: "encryptionkey@0"
I20260810 19:02:53.891350 28210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:53.908803 28210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:53.911449 28210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:53.911659 28210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:53.918794 28210 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36169
I20260810 19:02:53.918846 28333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36169 every 8 connection(s)
I20260810 19:02:53.919729 28210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:53.922343 28334 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:53.927625 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28210
I20260810 19:02:53.927773 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:53.927898 28334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112: Bootstrap starting.
I20260810 19:02:53.928097 28238 external_mini_cluster.cc:1428] Setting key f45487c4c95e165f88575eced8e1bb87
I20260810 19:02:53.930399 28334 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:53.931201 28334 log.cc:824] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:53.932973 28334 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112: No bootstrap required, opened a new log
I20260810 19:02:53.936424 28334 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36169 } }
Aug 10 19:02:53 dist-test-slave-znh6 krb5kdc[28187](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388573, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:02:53.936658 28334 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:53.936746 28334 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a5d8ce320de4fbe89f73da5d4e1b112, State: Initialized, Role: FOLLOWER
I20260810 19:02:53.937242 28334 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36169 } }
I20260810 19:02:53.937453 28334 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:53.937551 28334 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:53.937650 28334 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:53.938728 28334 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36169 } }
I20260810 19:02:53.939090 28334 leader_election.cc:302] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a5d8ce320de4fbe89f73da5d4e1b112; no voters:
I20260810 19:02:53.939392 28334 leader_election.cc:288] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:53.939499 28339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:53.939781 28339 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [term 1 LEADER]: Becoming Leader. State: Replica: 5a5d8ce320de4fbe89f73da5d4e1b112, State: Running, Role: LEADER
I20260810 19:02:53.940109 28339 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36169 } }
I20260810 19:02:53.940759 28334 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:53.941993 28341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36169 } } }
I20260810 19:02:53.942001 28342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a5d8ce320de4fbe89f73da5d4e1b112. Latest consensus state: current_term: 1 leader_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a5d8ce320de4fbe89f73da5d4e1b112" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36169 } } }
I20260810 19:02:53.942154 28341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:53.942154 28342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:53.942756 28354 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:53.943040 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:53.929458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52091 (local address 127.0.0.1:36169)
0810 19:02:53.929878 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:53.929892 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:53.929927 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:02:53.930599 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:53.930616 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:53.930971 (+ 355us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:53.931299 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:53.932172 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:53.933186 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:53.934058 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:53.934532 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:53.934709 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:53.937355 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:53.937361 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:53.937377 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:53.937404 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:53.939465 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:53.940077 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:53.940083 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:53.940088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:53.940306 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:53.940605 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:53.940609 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:53.940610 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:53.941118 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:53.941327 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:53.941665 (+ 338us) server_negotiation.cc:299] Negotiation successful
0810 19:02:53.941997 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":127,"threads_started":1}
I20260810 19:02:53.946615 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:02:53.946990 28354 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:53.952188 28354 catalog_manager.cc:1347] Generated new cluster ID: c3573fe0665b4076b7645a9863d56471
I20260810 19:02:53.952248 28354 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:53.960877 28337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:53.951319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51071 (local address 127.0.0.1:36169)
0810 19:02:53.951411 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:53.951415 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:53.952560 (+ 1145us) server_negotiation.cc:407] Connection header received
0810 19:02:53.952682 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:53.952686 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:53.952732 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:53.952808 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:53.953585 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:53.954371 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:53.955499 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:53.955877 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:53.955993 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:53.957450 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:53.957454 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:53.957457 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:53.957489 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:53.959141 (+ 1652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:53.959699 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:53.959703 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:53.959705 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:53.959790 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:53.960064 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:53.960067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:53.960069 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:53.960270 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:53.960420 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:53.960619 (+ 199us) server_negotiation.cc:299] Negotiation successful
0810 19:02:53.960729 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260810 19:02:53.974223 28354 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:53.975397 28354 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:53.985234 28354 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a5d8ce320de4fbe89f73da5d4e1b112: Generated new TSK 0
I20260810 19:02:53.985848 28354 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:53.994961 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28210
2026-08-10T19:02:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1634 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:54 dist-test-slave-znh6 krb5kdc[28378](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:54 dist-test-slave-znh6 krb5kdc[28378](info): set up 2 sockets
Aug 10 19:02:54 dist-test-slave-znh6 krb5kdc[28378](info): commencing operation
krb5kdc: starting...
W20260810 19:02:55.246585 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:55 dist-test-slave-znh6 krb5kdc[28378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388575, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:02:55.373540 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45367
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:55.485128 28396 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:55.485406 28396 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:55.485497 28396 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:55.489487 28396 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:55.489569 28396 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:55.489607 28396 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:55.489665 28396 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:55.489705 28396 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:55.494736 28396 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45367
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28396
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:55.495752 28396 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:55.496754 28396 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:55.505399 28411 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:55.505388 28405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:55.505574 28396 server_base.cc:1034] running on GCE node
W20260810 19:02:55.505388 28404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:55.506310 28396 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:55.507392 28396 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:55.508546 28396 hybrid_clock.cc:648] HybridClock initialized: now 1786388575508525 us; error 45 us; skew 500 ppm
Aug 10 19:02:55 dist-test-slave-znh6 krb5kdc[28378](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388575, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:55.511338 28396 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:02:55.512126 28396 webserver.cc:460] Webserver started at http://127.0.0.1:35145/ using document root <none> and password file <none>
I20260810 19:02:55.512616 28396 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:55.512673 28396 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:55.512843 28396 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:55.514735 28396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "98f12dae52584ab5a10ffc03e5c208ad"
format_stamp: "Formatted at 2026-08-10 19:02:55 on dist-test-slave-znh6"
server_key: "bdf492a2767b74324e4fcdf911b8d5ad"
server_key_iv: "d70673b38e437b03ad94abae4c8f2d04"
server_key_version: "encryptionkey@0"
I20260810 19:02:55.515380 28396 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "98f12dae52584ab5a10ffc03e5c208ad"
format_stamp: "Formatted at 2026-08-10 19:02:55 on dist-test-slave-znh6"
server_key: "bdf492a2767b74324e4fcdf911b8d5ad"
server_key_iv: "d70673b38e437b03ad94abae4c8f2d04"
server_key_version: "encryptionkey@0"
I20260810 19:02:55.518975 28396 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:02:55.521224 28421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:55.522454 28396 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:02:55.522605 28396 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "98f12dae52584ab5a10ffc03e5c208ad"
format_stamp: "Formatted at 2026-08-10 19:02:55 on dist-test-slave-znh6"
server_key: "bdf492a2767b74324e4fcdf911b8d5ad"
server_key_iv: "d70673b38e437b03ad94abae4c8f2d04"
server_key_version: "encryptionkey@0"
I20260810 19:02:55.522716 28396 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:55.535822 28396 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:55.538941 28396 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:55.539139 28396 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:55.546185 28396 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45641
I20260810 19:02:55.546242 28508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45641 every 8 connection(s)
I20260810 19:02:55.547173 28396 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:55.549216 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28396
I20260810 19:02:55.549389 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:55.549716 28238 external_mini_cluster.cc:1428] Setting key 97deb8885c515e186465e7d33b92ff87
I20260810 19:02:55.550141 28509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:55.556206 28509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad: Bootstrap starting.
I20260810 19:02:55.558444 28509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:55.558488 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:55.551163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34515 (local address 127.0.0.1:45641)
0810 19:02:55.551607 (+ 444us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:55.551657 (+ 50us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:55.551693 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:02:55.552318 (+ 625us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:55.552333 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:55.552637 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:55.552955 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:55.553380 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:55.554357 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:55.555358 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:55.555819 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:55.555997 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:55.556876 (+ 879us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:55.556881 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:55.556894 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:55.557249 (+ 355us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:55.557311 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:55.557333 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:55.557440 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:55.557614 (+ 174us) server_negotiation.cc:299] Negotiation successful
0810 19:02:55.557817 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":137,"threads_started":1}
I20260810 19:02:55.559405 28509 log.cc:824] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:55.561388 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:02:55.561511 28509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad: No bootstrap required, opened a new log
I20260810 19:02:55.565438 28509 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f12dae52584ab5a10ffc03e5c208ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260810 19:02:55.565645 28509 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:55.565681 28509 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98f12dae52584ab5a10ffc03e5c208ad, State: Initialized, Role: FOLLOWER
I20260810 19:02:55.566174 28509 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f12dae52584ab5a10ffc03e5c208ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260810 19:02:55.566329 28509 raft_consensus.cc:388] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:55.566403 28509 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:55.566502 28509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:55.567454 28509 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f12dae52584ab5a10ffc03e5c208ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260810 19:02:55.567809 28509 leader_election.cc:302] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 98f12dae52584ab5a10ffc03e5c208ad; no voters:
I20260810 19:02:55.568135 28509 leader_election.cc:288] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:55.568207 28522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:55.568401 28522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [term 1 LEADER]: Becoming Leader. State: Replica: 98f12dae52584ab5a10ffc03e5c208ad, State: Running, Role: LEADER
I20260810 19:02:55.568737 28522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f12dae52584ab5a10ffc03e5c208ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } }
I20260810 19:02:55.569380 28509 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:55.570266 28515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:55.563497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57261 (local address 127.0.0.1:45641)
0810 19:02:55.563612 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:55.563615 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:55.565981 (+ 2366us) server_negotiation.cc:407] Connection header received
0810 19:02:55.566104 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:55.566109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:55.566157 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:55.566245 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:55.566539 (+ 294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:55.567195 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:55.568148 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:55.568501 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:55.568991 (+ 490us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:55.569463 (+ 472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:55.569469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:55.569475 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:55.569489 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:55.569515 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:55.569527 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:55.569621 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:55.570021 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 19:02:55.570081 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":22,"server-negotiator.queue_time_us":46}
I20260810 19:02:55.570609 28523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "98f12dae52584ab5a10ffc03e5c208ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f12dae52584ab5a10ffc03e5c208ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } } }
I20260810 19:02:55.570794 28523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:55.570389 28524 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 98f12dae52584ab5a10ffc03e5c208ad. Latest consensus state: current_term: 1 leader_uuid: "98f12dae52584ab5a10ffc03e5c208ad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "98f12dae52584ab5a10ffc03e5c208ad" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45641 } } }
I20260810 19:02:55.571228 28524 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad [sys.catalog]: This master's current role is: LEADER
W20260810 19:02:55.574545 28545 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:02:55.574642 28545 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:02:55.574733 28539 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:55.575417 28539 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:55.580281 28539 catalog_manager.cc:1347] Generated new cluster ID: 919b53f36c13429098be60b459abbfa4
I20260810 19:02:55.580351 28539 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:55.592795 28539 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:55.593967 28539 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:55.601867 28539 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 98f12dae52584ab5a10ffc03e5c208ad: Generated new TSK 0
I20260810 19:02:55.602626 28539 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:55.608067 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28396
2026-08-10T19:02:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1612 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:55 dist-test-slave-znh6 krb5kdc[28557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:55 dist-test-slave-znh6 krb5kdc[28557](info): set up 2 sockets
Aug 10 19:02:55 dist-test-slave-znh6 krb5kdc[28557](info): commencing operation
krb5kdc: starting...
W20260810 19:02:56.864274 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.005s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:56 dist-test-slave-znh6 krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388576, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:02:56.987860 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41691
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41691
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:57.098971 28578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:57.099208 28578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:57.099248 28578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:57.103080 28578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:57.103124 28578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:57.103143 28578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:57.103159 28578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:57.103173 28578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:57.108158 28578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41691
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41691
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:57.109019 28578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:57.109915 28578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:57.118137 28588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:57.118144 28586 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:57.118177 28578 server_base.cc:1034] running on GCE node
W20260810 19:02:57.118144 28584 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:57.119007 28578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:57.120064 28578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:57.121237 28578 hybrid_clock.cc:648] HybridClock initialized: now 1786388577121218 us; error 31 us; skew 500 ppm
Aug 10 19:02:57 dist-test-slave-znh6 krb5kdc[28557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388577, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:57.124130 28578 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:02:57.124994 28578 webserver.cc:460] Webserver started at http://127.0.0.1:37499/ using document root <none> and password file <none>
I20260810 19:02:57.125540 28578 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:57.125634 28578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:57.125875 28578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:57.127909 28578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "19c593b69570458c860a0be6b16f1367"
format_stamp: "Formatted at 2026-08-10 19:02:57 on dist-test-slave-znh6"
server_key: "b2d4179d2e9cf5b85895f90528572f53"
server_key_iv: "1b3327d91eaeae60ad5a288606a8baea"
server_key_version: "encryptionkey@0"
I20260810 19:02:57.128470 28578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "19c593b69570458c860a0be6b16f1367"
format_stamp: "Formatted at 2026-08-10 19:02:57 on dist-test-slave-znh6"
server_key: "b2d4179d2e9cf5b85895f90528572f53"
server_key_iv: "1b3327d91eaeae60ad5a288606a8baea"
server_key_version: "encryptionkey@0"
I20260810 19:02:57.132187 28578 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:02:57.134536 28602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:57.135793 28578 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:02:57.135952 28578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "19c593b69570458c860a0be6b16f1367"
format_stamp: "Formatted at 2026-08-10 19:02:57 on dist-test-slave-znh6"
server_key: "b2d4179d2e9cf5b85895f90528572f53"
server_key_iv: "1b3327d91eaeae60ad5a288606a8baea"
server_key_version: "encryptionkey@0"
I20260810 19:02:57.136063 28578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:57.140439 28578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:57.143230 28578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:57.143445 28578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:57.150439 28578 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41691
I20260810 19:02:57.150452 28692 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41691 every 8 connection(s)
I20260810 19:02:57.151329 28578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:57.153407 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28578
I20260810 19:02:57.153548 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:57.153863 28238 external_mini_cluster.cc:1428] Setting key 98fe3db704b6df9272bfd32f027d0579
I20260810 19:02:57.154358 28693 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:57.158932 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:57.155207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35783 (local address 127.0.0.1:41691)
0810 19:02:57.155617 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:57.155641 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:57.155676 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:02:57.156328 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:57.156344 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:57.156601 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:57.156867 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:57.157671 (+ 804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:57.157676 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:57.157687 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:57.157696 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:57.157751 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:57.157773 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:57.157880 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:57.158027 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 19:02:57.158191 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":271,"thread_start_us":138,"threads_started":1}
I20260810 19:02:57.161064 28693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367: Bootstrap starting.
I20260810 19:02:57.161828 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:02:57.163867 28693 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:57.164793 28693 log.cc:824] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:57.165748 28697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:57.164116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33243 (local address 127.0.0.1:41691)
0810 19:02:57.164237 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:57.164250 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:57.164413 (+ 163us) server_negotiation.cc:407] Connection header received
0810 19:02:57.164504 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:57.164506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:57.164540 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:57.164633 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:57.165315 (+ 682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:57.165327 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:57.165328 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:02:57.165339 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:57.165361 (+ 22us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:02:57.165370 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:57.165438 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:57.165567 (+ 129us) server_negotiation.cc:299] Negotiation successful
0810 19:02:57.165594 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 19:02:57.167035 28693 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367: No bootstrap required, opened a new log
I20260810 19:02:57.170337 28693 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c593b69570458c860a0be6b16f1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
I20260810 19:02:57.170569 28693 raft_consensus.cc:374] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:57.170670 28693 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19c593b69570458c860a0be6b16f1367, State: Initialized, Role: FOLLOWER
I20260810 19:02:57.171162 28693 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c593b69570458c860a0be6b16f1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
I20260810 19:02:57.171319 28693 raft_consensus.cc:388] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:57.171387 28693 raft_consensus.cc:482] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:57.171482 28693 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:57.172297 28693 raft_consensus.cc:504] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c593b69570458c860a0be6b16f1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
I20260810 19:02:57.172631 28693 leader_election.cc:302] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 19c593b69570458c860a0be6b16f1367; no voters:
I20260810 19:02:57.172933 28693 leader_election.cc:288] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:57.173095 28705 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:57.173377 28705 raft_consensus.cc:686] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [term 1 LEADER]: Becoming Leader. State: Replica: 19c593b69570458c860a0be6b16f1367, State: Running, Role: LEADER
I20260810 19:02:57.173684 28705 consensus_queue.cc:237] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c593b69570458c860a0be6b16f1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } }
I20260810 19:02:57.174120 28693 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:57.175330 28709 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 19c593b69570458c860a0be6b16f1367. Latest consensus state: current_term: 1 leader_uuid: "19c593b69570458c860a0be6b16f1367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c593b69570458c860a0be6b16f1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } } }
I20260810 19:02:57.175393 28708 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "19c593b69570458c860a0be6b16f1367" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "19c593b69570458c860a0be6b16f1367" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41691 } } }
I20260810 19:02:57.175470 28709 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:57.175470 28708 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:57.175863 28718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:57.179014 28718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:57.184216 28718 catalog_manager.cc:1347] Generated new cluster ID: e8d37eb3e3cf4fb79b2f632c46d68063
I20260810 19:02:57.184310 28718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:57.209331 28718 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:57.209712 28718 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:57.215654 28718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 19c593b69570458c860a0be6b16f1367: Generated new TSK 0
I20260810 19:02:57.216370 28718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:57.231074 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28578
2026-08-10T19:02:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1625 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:57 dist-test-slave-znh6 krb5kdc[28742](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:57 dist-test-slave-znh6 krb5kdc[28742](info): set up 2 sockets
Aug 10 19:02:57 dist-test-slave-znh6 krb5kdc[28742](info): commencing operation
krb5kdc: starting...
W20260810 19:02:58.492163 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.003s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:02:58 dist-test-slave-znh6 krb5kdc[28742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:02:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:02:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:02:58.613572 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35547
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38851
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:02:58.725003 28763 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:02:58.725274 28763 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:02:58.725351 28763 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:02:58.729307 28763 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:02:58.729389 28763 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:02:58.729429 28763 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:02:58.729487 28763 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:02:58.729527 28763 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:02:58.734552 28763 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38851
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35547
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28763
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:02:58.735531 28763 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:02:58.736481 28763 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:02:58.744706 28771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:58.744782 28772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:02:58.744747 28776 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:02:58.745067 28763 server_base.cc:1034] running on GCE node
I20260810 19:02:58.745649 28763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:02:58.746990 28763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:02:58.748175 28763 hybrid_clock.cc:648] HybridClock initialized: now 1786388578748156 us; error 31 us; skew 500 ppm
Aug 10 19:02:58 dist-test-slave-znh6 krb5kdc[28742](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388578, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:02:58.750953 28763 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:02:58.751792 28763 webserver.cc:460] Webserver started at http://127.0.0.1:34209/ using document root <none> and password file <none>
I20260810 19:02:58.752282 28763 fs_manager.cc:359] Metadata directory not provided
I20260810 19:02:58.752343 28763 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:02:58.752499 28763 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:02:58.754355 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "3d5449d5aa204a309b9dcc081d3bdb50"
format_stamp: "Formatted at 2026-08-10 19:02:58 on dist-test-slave-znh6"
server_key: "aecb71437c6ff404671c69e12ff16d38"
server_key_iv: "ca77aaea3175c8e55abf32e77aeb33b2"
server_key_version: "encryptionkey@0"
I20260810 19:02:58.754935 28763 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "3d5449d5aa204a309b9dcc081d3bdb50"
format_stamp: "Formatted at 2026-08-10 19:02:58 on dist-test-slave-znh6"
server_key: "aecb71437c6ff404671c69e12ff16d38"
server_key_iv: "ca77aaea3175c8e55abf32e77aeb33b2"
server_key_version: "encryptionkey@0"
I20260810 19:02:58.758383 28763 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:02:58.760634 28789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:02:58.761816 28763 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260810 19:02:58.761958 28763 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "3d5449d5aa204a309b9dcc081d3bdb50"
format_stamp: "Formatted at 2026-08-10 19:02:58 on dist-test-slave-znh6"
server_key: "aecb71437c6ff404671c69e12ff16d38"
server_key_iv: "ca77aaea3175c8e55abf32e77aeb33b2"
server_key_version: "encryptionkey@0"
I20260810 19:02:58.762070 28763 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:02:58.777449 28763 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:02:58.780586 28763 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:02:58.780786 28763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:02:58.788293 28763 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35547
I20260810 19:02:58.788322 28876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35547 every 8 connection(s)
I20260810 19:02:58.789191 28763 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:02:58.791893 28877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:02:58.797088 28877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50: Bootstrap starting.
I20260810 19:02:58.799348 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28763
I20260810 19:02:58.799511 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:02:58.799646 28877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50: Neither blocks nor log segments found. Creating new log.
I20260810 19:02:58.799867 28238 external_mini_cluster.cc:1428] Setting key 84e15b695645de2e4d3643cb05db4712
I20260810 19:02:58.800480 28877 log.cc:824] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50: Log is configured to *not* fsync() on all Append() calls
I20260810 19:02:58.802702 28877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50: No bootstrap required, opened a new log
I20260810 19:02:58.807049 28877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260810 19:02:58.807324 28877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:02:58.807397 28877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d5449d5aa204a309b9dcc081d3bdb50, State: Initialized, Role: FOLLOWER
I20260810 19:02:58.807891 28877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260810 19:02:58.808032 28877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:02:58.808128 28877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:02:58.808266 28877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:02:58.809294 28877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
Aug 10 19:02:58 dist-test-slave-znh6 krb5kdc[28742](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388578, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:02:58.809670 28877 leader_election.cc:302] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3d5449d5aa204a309b9dcc081d3bdb50; no voters:
I20260810 19:02:58.810041 28877 leader_election.cc:288] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:02:58.810133 28885 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:02:58.810352 28885 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [term 1 LEADER]: Becoming Leader. State: Replica: 3d5449d5aa204a309b9dcc081d3bdb50, State: Running, Role: LEADER
I20260810 19:02:58.810657 28885 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } }
I20260810 19:02:58.811470 28877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:02:58.812254 28890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3d5449d5aa204a309b9dcc081d3bdb50. Latest consensus state: current_term: 1 leader_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } } }
I20260810 19:02:58.812393 28890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:58.812718 28886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3d5449d5aa204a309b9dcc081d3bdb50" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35547 } } }
I20260810 19:02:58.812820 28886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50 [sys.catalog]: This master's current role is: LEADER
I20260810 19:02:58.813107 28906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:02:58.816252 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:58.801058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:50027 (local address 127.0.0.1:35547)
0810 19:02:58.801521 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:58.801540 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:58.801582 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:02:58.802401 (+ 819us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:58.802424 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:58.802863 (+ 439us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:58.803213 (+ 350us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:58.803934 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:58.805431 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:58.806367 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:58.807073 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:58.807248 (+ 175us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:58.810301 (+ 3053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:58.810307 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:58.810324 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:58.810353 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:58.812575 (+ 2222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:58.813268 (+ 693us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:58.813278 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:58.813287 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:58.813445 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:58.813924 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:58.813927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:58.813928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:58.814446 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:58.814727 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:58.815096 (+ 369us) server_negotiation.cc:299] Negotiation successful
0810 19:02:58.815341 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":128,"threads_started":1}
I20260810 19:02:58.816308 28906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:02:58.819617 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:02:58.822413 28906 catalog_manager.cc:1347] Generated new cluster ID: 4a1d315493bf4a4880f2b24ab571921c
I20260810 19:02:58.822480 28906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:02:58.830477 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:02:58.821901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:39101 (local address 127.0.0.1:35547)
0810 19:02:58.821998 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 19:02:58.822002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:02:58.822095 (+ 93us) server_negotiation.cc:407] Connection header received
0810 19:02:58.822209 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:02:58.822212 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:02:58.822255 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:02:58.822347 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:02:58.823269 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:58.824034 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:58.825111 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:02:58.825436 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:02:58.825497 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:02:58.827151 (+ 1654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:02:58.827155 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:02:58.827157 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:02:58.827180 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:02:58.828691 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:58.829195 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:58.829198 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:58.829200 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:58.829270 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:02:58.829588 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:02:58.829591 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:02:58.829593 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:02:58.829762 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:02:58.829863 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:02:58.830241 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 19:02:58.830330 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 19:02:58.833966 28906 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:02:58.834777 28906 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:02:58.842525 28906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3d5449d5aa204a309b9dcc081d3bdb50: Generated new TSK 0
I20260810 19:02:58.843324 28906 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:02:58.852201 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28763
W20260810 19:02:58.866977 28740 connection.cc:535] client connection to 127.0.0.1:35547 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35547): Connection reset by peer (error 104)
2026-08-10T19:02:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1617 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:02:58 dist-test-slave-znh6 krb5kdc[28936](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:02:58 dist-test-slave-znh6 krb5kdc[28936](info): set up 2 sockets
Aug 10 19:02:58 dist-test-slave-znh6 krb5kdc[28936](info): commencing operation
krb5kdc: starting...
W20260810 19:03:00.102205 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.221s user 0.002s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:00 dist-test-slave-znh6 krb5kdc[28936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388580, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:03:00.230195 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45393
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34689
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45393
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:00.343519 28953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:00.343747 28953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:00.343787 28953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:00.347838 28953 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:00.347894 28953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:00.347915 28953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:00.347931 28953 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:00.347946 28953 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:00.353005 28953 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34689
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45393
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:00.353966 28953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:00.354955 28953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:00.363296 28961 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:00.363277 28962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:00.363554 28953 server_base.cc:1034] running on GCE node
W20260810 19:03:00.363343 28966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:00.364293 28953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:00.365515 28953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:00.366678 28953 hybrid_clock.cc:648] HybridClock initialized: now 1786388580366659 us; error 29 us; skew 500 ppm
Aug 10 19:03:00 dist-test-slave-znh6 krb5kdc[28936](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388580, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:00.369324 28953 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:03:00.370148 28953 webserver.cc:460] Webserver started at http://127.0.0.1:39393/ using document root <none> and password file <none>
I20260810 19:03:00.370676 28953 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:00.370735 28953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:00.371016 28953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:00.372937 28953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "3b4ccf885bad4aabaefc196d5089448c"
format_stamp: "Formatted at 2026-08-10 19:03:00 on dist-test-slave-znh6"
server_key: "9821c21cabf7e1bc452ce756d6bb12d8"
server_key_iv: "d226388256c17cf3154579ced36bd0b2"
server_key_version: "encryptionkey@0"
I20260810 19:03:00.373517 28953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "3b4ccf885bad4aabaefc196d5089448c"
format_stamp: "Formatted at 2026-08-10 19:03:00 on dist-test-slave-znh6"
server_key: "9821c21cabf7e1bc452ce756d6bb12d8"
server_key_iv: "d226388256c17cf3154579ced36bd0b2"
server_key_version: "encryptionkey@0"
I20260810 19:03:00.377543 28953 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 19:03:00.380553 28976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:00.381759 28953 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:03:00.381899 28953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "3b4ccf885bad4aabaefc196d5089448c"
format_stamp: "Formatted at 2026-08-10 19:03:00 on dist-test-slave-znh6"
server_key: "9821c21cabf7e1bc452ce756d6bb12d8"
server_key_iv: "d226388256c17cf3154579ced36bd0b2"
server_key_version: "encryptionkey@0"
I20260810 19:03:00.382010 28953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:00.404933 28953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:00.408851 28953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:00.409053 28953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:00.416432 28953 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45393
I20260810 19:03:00.416437 29054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45393 every 8 connection(s)
I20260810 19:03:00.417382 28953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:00.420113 29056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:00.425669 29056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c: Bootstrap starting.
I20260810 19:03:00.426731 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 28953
I20260810 19:03:00.426914 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:00.427194 28238 external_mini_cluster.cc:1428] Setting key b20be83681ddcb966f06cd7cfc9138f2
I20260810 19:03:00.428350 29056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:00.429104 29056 log.cc:824] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:00.431048 29056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c: No bootstrap required, opened a new log
I20260810 19:03:00.434510 29056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4ccf885bad4aabaefc196d5089448c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260810 19:03:00.434705 29056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:00.434737 29056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b4ccf885bad4aabaefc196d5089448c, State: Initialized, Role: FOLLOWER
I20260810 19:03:00.435295 29056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4ccf885bad4aabaefc196d5089448c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260810 19:03:00.435454 29056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:00.435545 29056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:00.435648 29056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:00.436089 29059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:00.428253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:57473 (local address 127.0.0.1:45393)
0810 19:03:00.428692 (+ 439us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:00.428707 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:00.428740 (+ 33us) server_negotiation.cc:407] Connection header received
0810 19:03:00.429436 (+ 696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:00.429479 (+ 43us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:00.429914 (+ 435us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:00.430241 (+ 327us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:00.430636 (+ 395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:00.431723 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:00.432657 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:00.433133 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:00.433313 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:00.434233 (+ 920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:00.434238 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:00.434250 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:00.434609 (+ 359us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:00.434658 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:00.434680 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:00.434915 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:00.435108 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 19:03:00.435244 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":144,"threads_started":1}
I20260810 19:03:00.436563 29056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4ccf885bad4aabaefc196d5089448c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260810 19:03:00.436902 29056 leader_election.cc:302] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b4ccf885bad4aabaefc196d5089448c; no voters:
I20260810 19:03:00.437238 29056 leader_election.cc:288] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:00.437335 29061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:00.437515 29061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [term 1 LEADER]: Becoming Leader. State: Replica: 3b4ccf885bad4aabaefc196d5089448c, State: Running, Role: LEADER
I20260810 19:03:00.437796 29061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4ccf885bad4aabaefc196d5089448c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } }
I20260810 19:03:00.438465 29056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:00.438903 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:00.438994 29062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b4ccf885bad4aabaefc196d5089448c. Latest consensus state: current_term: 1 leader_uuid: "3b4ccf885bad4aabaefc196d5089448c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4ccf885bad4aabaefc196d5089448c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } } }
I20260810 19:03:00.439101 29062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:00.439527 29071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:00.439826 29064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b4ccf885bad4aabaefc196d5089448c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b4ccf885bad4aabaefc196d5089448c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45393 } } }
I20260810 19:03:00.439935 29064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:00.442765 29071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:00.448767 29071 catalog_manager.cc:1347] Generated new cluster ID: d1f9904e0e19444c80e54180d2b2457a
I20260810 19:03:00.448825 29071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:00.451634 29059 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:00.445081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:57155 (local address 127.0.0.1:45393)
0810 19:03:00.445197 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:00.445201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:00.445565 (+ 364us) server_negotiation.cc:407] Connection header received
0810 19:03:00.445735 (+ 170us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:00.445738 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:00.445793 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:00.445843 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:00.446226 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:00.446875 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:00.447886 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:00.448147 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:00.448261 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:00.451182 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:00.451187 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:00.451189 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:00.451204 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:00.451236 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:00.451250 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:00.451349 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:00.451465 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 19:03:00.451503 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:03:00.456699 29071 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:00.457792 29071 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:00.475533 29071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b4ccf885bad4aabaefc196d5089448c: Generated new TSK 0
I20260810 19:03:00.476310 29071 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:00.488000 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 28953
W20260810 19:03:00.503494 28934 connection.cc:535] client connection to 127.0.0.1:45393 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45393): Connection reset by peer (error 104)
2026-08-10T19:03:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1636 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:00 dist-test-slave-znh6 krb5kdc[29101](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:00 dist-test-slave-znh6 krb5kdc[29101](info): set up 2 sockets
Aug 10 19:03:00 dist-test-slave-znh6 krb5kdc[29101](info): commencing operation
krb5kdc: starting...
W20260810 19:03:01.753075 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.235s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:01 dist-test-slave-znh6 krb5kdc[29101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388581, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:03:01.877475 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36585
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39717
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:01.987777 29127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:01.988008 29127 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:01.988045 29127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:01.991947 29127 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:01.991997 29127 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:01.992017 29127 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:01.992033 29127 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:01.992079 29127 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:01.997121 29127 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39717
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36585
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29127
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:01.998050 29127 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:01.999065 29127 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:02.007707 29133 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:02.007800 29127 server_base.cc:1034] running on GCE node
W20260810 19:03:02.007700 29136 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:02.007700 29134 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:02.008534 29127 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:02.009543 29127 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:02.010737 29127 hybrid_clock.cc:648] HybridClock initialized: now 1786388582010711 us; error 40 us; skew 500 ppm
Aug 10 19:03:02 dist-test-slave-znh6 krb5kdc[29101](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388582, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:02.013456 29127 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:03:02.014350 29127 webserver.cc:460] Webserver started at http://127.0.0.1:34221/ using document root <none> and password file <none>
I20260810 19:03:02.014950 29127 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:02.015043 29127 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:02.015271 29127 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:02.017208 29127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "89892537be1d407bb860dcd0916532be"
format_stamp: "Formatted at 2026-08-10 19:03:02 on dist-test-slave-znh6"
server_key: "d7ef9a86431b50950c77f1bb1fb98121"
server_key_iv: "bcb68c2288a63aef3710ca85b403df05"
server_key_version: "encryptionkey@0"
I20260810 19:03:02.017788 29127 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "89892537be1d407bb860dcd0916532be"
format_stamp: "Formatted at 2026-08-10 19:03:02 on dist-test-slave-znh6"
server_key: "d7ef9a86431b50950c77f1bb1fb98121"
server_key_iv: "bcb68c2288a63aef3710ca85b403df05"
server_key_version: "encryptionkey@0"
I20260810 19:03:02.021381 29127 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.006s sys 0.000s
I20260810 19:03:02.023875 29150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:02.025022 29127 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:02.025166 29127 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "89892537be1d407bb860dcd0916532be"
format_stamp: "Formatted at 2026-08-10 19:03:02 on dist-test-slave-znh6"
server_key: "d7ef9a86431b50950c77f1bb1fb98121"
server_key_iv: "bcb68c2288a63aef3710ca85b403df05"
server_key_version: "encryptionkey@0"
I20260810 19:03:02.025274 29127 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:02.029909 29127 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:02.032711 29127 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:02.032898 29127 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:02.040709 29127 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36585
I20260810 19:03:02.040764 29223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36585 every 8 connection(s)
I20260810 19:03:02.041600 29127 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:02.043498 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29127
I20260810 19:03:02.043663 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:02.043974 28238 external_mini_cluster.cc:1428] Setting key fdc5b0ac69317abf265ddb913593ab0b
I20260810 19:03:02.044646 29225 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:02.049197 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:02.045321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:48167 (local address 127.0.0.1:36585)
0810 19:03:02.045713 (+ 392us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:02.045728 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:02.045763 (+ 35us) server_negotiation.cc:407] Connection header received
0810 19:03:02.046365 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:02.046380 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:02.046649 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:02.047017 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:02.047848 (+ 831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:02.047853 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:02.047867 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:02.047879 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:02.047943 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:02.047968 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:02.048105 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:02.048217 (+ 112us) server_negotiation.cc:299] Negotiation successful
0810 19:03:02.048361 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":110,"threads_started":1}
I20260810 19:03:02.052165 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:02.052196 29225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be: Bootstrap starting.
I20260810 19:03:02.054627 29225 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:02.055428 29225 log.cc:824] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:02.055982 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:02.054163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:36421 (local address 127.0.0.1:36585)
0810 19:03:02.054270 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:02.054273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:02.054546 (+ 273us) server_negotiation.cc:407] Connection header received
0810 19:03:02.054674 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:02.054676 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:02.054728 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:02.054849 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:02.055547 (+ 698us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:02.055552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:02.055555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:02.055568 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:02.055594 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:02.055606 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:02.055700 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:02.055798 (+ 98us) server_negotiation.cc:299] Negotiation successful
0810 19:03:02.055834 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 19:03:02.057410 29225 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be: No bootstrap required, opened a new log
I20260810 19:03:02.060773 29225 raft_consensus.cc:348] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89892537be1d407bb860dcd0916532be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260810 19:03:02.060978 29225 raft_consensus.cc:374] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:02.061023 29225 raft_consensus.cc:729] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 89892537be1d407bb860dcd0916532be, State: Initialized, Role: FOLLOWER
I20260810 19:03:02.061463 29225 consensus_queue.cc:260] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89892537be1d407bb860dcd0916532be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260810 19:03:02.061622 29225 raft_consensus.cc:388] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:02.061699 29225 raft_consensus.cc:482] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:02.061802 29225 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:02.062589 29225 raft_consensus.cc:504] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89892537be1d407bb860dcd0916532be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260810 19:03:02.062930 29225 leader_election.cc:302] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 89892537be1d407bb860dcd0916532be; no voters:
I20260810 19:03:02.063184 29225 leader_election.cc:288] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:02.063302 29240 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:02.063597 29240 raft_consensus.cc:686] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [term 1 LEADER]: Becoming Leader. State: Replica: 89892537be1d407bb860dcd0916532be, State: Running, Role: LEADER
I20260810 19:03:02.063889 29240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89892537be1d407bb860dcd0916532be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } }
I20260810 19:03:02.064409 29225 sys_catalog.cc:564] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:02.065526 29241 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "89892537be1d407bb860dcd0916532be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89892537be1d407bb860dcd0916532be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } } }
I20260810 19:03:02.065660 29241 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:02.065989 29242 sys_catalog.cc:455] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [sys.catalog]: SysCatalogTable state changed. Reason: New leader 89892537be1d407bb860dcd0916532be. Latest consensus state: current_term: 1 leader_uuid: "89892537be1d407bb860dcd0916532be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "89892537be1d407bb860dcd0916532be" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36585 } } }
I20260810 19:03:02.066066 29261 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:02.066102 29242 sys_catalog.cc:458] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:02.069330 29261 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:02.074661 29261 catalog_manager.cc:1347] Generated new cluster ID: c85d362bca4a4aab818c2ca9e1df0a5d
I20260810 19:03:02.074744 29261 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:02.094551 29261 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:02.094894 29261 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:02.104305 29261 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 89892537be1d407bb860dcd0916532be: Generated new TSK 0
I20260810 19:03:02.104930 29261 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:02.112174 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29127
2026-08-10T19:03:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1622 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:02 dist-test-slave-znh6 krb5kdc[29275](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:02 dist-test-slave-znh6 krb5kdc[29275](info): set up 2 sockets
Aug 10 19:03:02 dist-test-slave-znh6 krb5kdc[29275](info): commencing operation
krb5kdc: starting...
W20260810 19:03:03.363261 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.222s user 0.006s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:03 dist-test-slave-znh6 krb5kdc[29275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:03:03.493913 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40231
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37573
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:03.605232 29297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:03.605480 29297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:03.605520 29297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:03.609516 29297 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:03.609572 29297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:03.609592 29297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:03.609647 29297 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:03.609679 29297 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:03.614790 29297 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37573
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40231
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:03.615867 29297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:03.616883 29297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:03:03.625586 29297 server_base.cc:1034] running on GCE node
W20260810 19:03:03.625563 29305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:03.625551 29307 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:03.625545 29309 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:03.626297 29297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:03.627470 29297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:03.628664 29297 hybrid_clock.cc:648] HybridClock initialized: now 1786388583628632 us; error 43 us; skew 500 ppm
Aug 10 19:03:03 dist-test-slave-znh6 krb5kdc[29275](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388583, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:03.636636 29297 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:03:03.637657 29297 webserver.cc:460] Webserver started at http://127.0.0.1:46577/ using document root <none> and password file <none>
I20260810 19:03:03.638266 29297 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:03.638363 29297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:03.638618 29297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:03.640758 29297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "30f8bf506a79427099e41cd81c7926e1"
format_stamp: "Formatted at 2026-08-10 19:03:03 on dist-test-slave-znh6"
server_key: "65398ff33e048be7b506b4e69554c86f"
server_key_iv: "50133be1c5f3c06de8a54d4a3b278f9b"
server_key_version: "encryptionkey@0"
I20260810 19:03:03.641355 29297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "30f8bf506a79427099e41cd81c7926e1"
format_stamp: "Formatted at 2026-08-10 19:03:03 on dist-test-slave-znh6"
server_key: "65398ff33e048be7b506b4e69554c86f"
server_key_iv: "50133be1c5f3c06de8a54d4a3b278f9b"
server_key_version: "encryptionkey@0"
I20260810 19:03:03.645267 29297 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 19:03:03.647820 29319 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:03.649062 29297 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:03.649211 29297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "30f8bf506a79427099e41cd81c7926e1"
format_stamp: "Formatted at 2026-08-10 19:03:03 on dist-test-slave-znh6"
server_key: "65398ff33e048be7b506b4e69554c86f"
server_key_iv: "50133be1c5f3c06de8a54d4a3b278f9b"
server_key_version: "encryptionkey@0"
I20260810 19:03:03.649325 29297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:03.672797 29297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:03.675602 29297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:03.675818 29297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:03.683364 29297 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40231
I20260810 19:03:03.683394 29395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40231 every 8 connection(s)
I20260810 19:03:03.684300 29297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:03.687103 29396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:03.689824 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29297
I20260810 19:03:03.689958 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:03.690250 28238 external_mini_cluster.cc:1428] Setting key 4f13a5d9142ea1cd9f2c9eccbf7ee245
I20260810 19:03:03.694000 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1: Bootstrap starting.
I20260810 19:03:03.696650 29396 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:03.697566 29396 log.cc:824] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:03.699761 29396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1: No bootstrap required, opened a new log
Aug 10 19:03:03 dist-test-slave-znh6 krb5kdc[29275](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388583, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 19:03:03.703717 29396 raft_consensus.cc:348] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f8bf506a79427099e41cd81c7926e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260810 19:03:03.703980 29396 raft_consensus.cc:374] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:03.704073 29396 raft_consensus.cc:729] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 30f8bf506a79427099e41cd81c7926e1, State: Initialized, Role: FOLLOWER
I20260810 19:03:03.704581 29396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f8bf506a79427099e41cd81c7926e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260810 19:03:03.704715 29396 raft_consensus.cc:388] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:03.704805 29396 raft_consensus.cc:482] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:03.704922 29396 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:03.705551 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:03.691670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:54985 (local address 127.0.0.1:40231)
0810 19:03:03.692099 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:03.692118 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:03.692160 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:03:03.692964 (+ 804us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:03.692990 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:03.693408 (+ 418us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:03.693771 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:03.694534 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:03.696029 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:03.696941 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:03.697648 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:03.697804 (+ 156us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:03.700579 (+ 2775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:03.700584 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:03.700597 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:03.700622 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:03.702314 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:03.702924 (+ 610us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:03.702929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:03.702934 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:03.703044 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:03.703442 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:03.703445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:03.703446 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:03.703854 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:03.704097 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:03.704475 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 19:03:03.704746 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":284,"thread_start_us":112,"threads_started":1}
I20260810 19:03:03.705814 29396 raft_consensus.cc:504] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f8bf506a79427099e41cd81c7926e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260810 19:03:03.706164 29396 leader_election.cc:302] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 30f8bf506a79427099e41cd81c7926e1; no voters:
I20260810 19:03:03.706461 29396 leader_election.cc:288] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:03.706560 29404 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:03.706784 29404 raft_consensus.cc:686] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [term 1 LEADER]: Becoming Leader. State: Replica: 30f8bf506a79427099e41cd81c7926e1, State: Running, Role: LEADER
I20260810 19:03:03.707150 29404 consensus_queue.cc:237] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f8bf506a79427099e41cd81c7926e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } }
I20260810 19:03:03.707748 29396 sys_catalog.cc:564] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:03.708979 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:03.708940 29406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "30f8bf506a79427099e41cd81c7926e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f8bf506a79427099e41cd81c7926e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } } }
I20260810 19:03:03.709065 29406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:03.709517 29418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:03.710004 29407 sys_catalog.cc:455] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 30f8bf506a79427099e41cd81c7926e1. Latest consensus state: current_term: 1 leader_uuid: "30f8bf506a79427099e41cd81c7926e1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "30f8bf506a79427099e41cd81c7926e1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40231 } } }
I20260810 19:03:03.710103 29407 sys_catalog.cc:458] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:03.712870 29418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:03.719708 29418 catalog_manager.cc:1347] Generated new cluster ID: 0aaa9b98917c4939bb3cbac4f61fecfd
I20260810 19:03:03.719822 29418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:03.720781 29399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:03.711520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:57025 (local address 127.0.0.1:40231)
0810 19:03:03.711628 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:03.711632 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:03.712012 (+ 380us) server_negotiation.cc:407] Connection header received
0810 19:03:03.712192 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:03.712196 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:03.712243 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:03.712325 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:03.713128 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:03.713842 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:03.715199 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:03.715568 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:03.715682 (+ 114us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:03.717268 (+ 1586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:03.717272 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:03.717274 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:03.717301 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:03.718687 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:03.719227 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:03.719231 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:03.719232 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:03.719318 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:03.719719 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:03.719725 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:03.719730 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:03.719964 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:03.720083 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:03.720528 (+ 445us) server_negotiation.cc:299] Negotiation successful
0810 19:03:03.720633 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 19:03:03.729822 29418 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:03.730995 29418 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:03.735522 29418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 30f8bf506a79427099e41cd81c7926e1: Generated new TSK 0
I20260810 19:03:03.736146 29418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:03.748940 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29297
W20260810 19:03:03.765825 29273 connection.cc:535] client connection to 127.0.0.1:40231 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40231): Connection reset by peer (error 104)
2026-08-10T19:03:03Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1639 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:03 dist-test-slave-znh6 krb5kdc[29447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:03 dist-test-slave-znh6 krb5kdc[29447](info): set up 2 sockets
Aug 10 19:03:03 dist-test-slave-znh6 krb5kdc[29447](info): commencing operation
krb5kdc: starting...
W20260810 19:03:05.007812 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.227s user 0.002s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:05 dist-test-slave-znh6 krb5kdc[29447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388585, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:03:05.133885 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41177
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46559
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41177
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:05.246178 29464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:05.246475 29464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:05.246562 29464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:05.250573 29464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:05.250665 29464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:05.250703 29464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:05.250741 29464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:05.250777 29464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:05.255940 29464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46559
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41177
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:05.256942 29464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:05.257931 29464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:05.266323 29470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:05.266332 29473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:05.266438 29471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:05.266410 29464 server_base.cc:1034] running on GCE node
I20260810 19:03:05.267235 29464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:05.268246 29464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:05.269404 29464 hybrid_clock.cc:648] HybridClock initialized: now 1786388585269386 us; error 31 us; skew 500 ppm
Aug 10 19:03:05 dist-test-slave-znh6 krb5kdc[29447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388585, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:05.272122 29464 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:03:05.272908 29464 webserver.cc:460] Webserver started at http://127.0.0.1:39775/ using document root <none> and password file <none>
I20260810 19:03:05.273407 29464 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:05.273471 29464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:05.273634 29464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:05.275640 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "8f082868a6f84966b038df4cfe7e1665"
format_stamp: "Formatted at 2026-08-10 19:03:05 on dist-test-slave-znh6"
server_key: "8f569172a85b36a207c9d2a06d8c6fd7"
server_key_iv: "bc67988f66c21e325515ba38b770e513"
server_key_version: "encryptionkey@0"
I20260810 19:03:05.276147 29464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "8f082868a6f84966b038df4cfe7e1665"
format_stamp: "Formatted at 2026-08-10 19:03:05 on dist-test-slave-znh6"
server_key: "8f569172a85b36a207c9d2a06d8c6fd7"
server_key_iv: "bc67988f66c21e325515ba38b770e513"
server_key_version: "encryptionkey@0"
I20260810 19:03:05.279654 29464 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:03:05.281869 29485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:05.283119 29464 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:05.283219 29464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "8f082868a6f84966b038df4cfe7e1665"
format_stamp: "Formatted at 2026-08-10 19:03:05 on dist-test-slave-znh6"
server_key: "8f569172a85b36a207c9d2a06d8c6fd7"
server_key_iv: "bc67988f66c21e325515ba38b770e513"
server_key_version: "encryptionkey@0"
I20260810 19:03:05.283322 29464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:05.298583 29464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:05.301780 29464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:05.301932 29464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:05.309226 29464 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41177
I20260810 19:03:05.309249 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41177 every 8 connection(s)
I20260810 19:03:05.310140 29464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:05.312870 29576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:05.318281 29576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665: Bootstrap starting.
I20260810 19:03:05.320070 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29464
I20260810 19:03:05.320217 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:05.320514 28238 external_mini_cluster.cc:1428] Setting key a57cbb5882711c882de3f88a47a645fd
I20260810 19:03:05.321005 29576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:05.321894 29576 log.cc:824] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:05.323899 29576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665: No bootstrap required, opened a new log
I20260810 19:03:05.327267 29576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f082868a6f84966b038df4cfe7e1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260810 19:03:05.327510 29576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:05.327591 29576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f082868a6f84966b038df4cfe7e1665, State: Initialized, Role: FOLLOWER
I20260810 19:03:05.328051 29576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f082868a6f84966b038df4cfe7e1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260810 19:03:05.328204 29576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:05.328289 29576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:05.328398 29576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:05.329304 29576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f082868a6f84966b038df4cfe7e1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260810 19:03:05.329579 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:05.321674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:42189 (local address 127.0.0.1:41177)
0810 19:03:05.322174 (+ 500us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:05.322191 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:05.322248 (+ 57us) server_negotiation.cc:407] Connection header received
0810 19:03:05.323103 (+ 855us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:05.323124 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:05.323476 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:05.323808 (+ 332us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:05.324233 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:05.325163 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:05.326090 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:05.326564 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:05.326736 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:05.327707 (+ 971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:05.327713 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:05.327729 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:05.328120 (+ 391us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:05.328174 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:05.328225 (+ 51us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:05.328387 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:05.328653 (+ 266us) server_negotiation.cc:299] Negotiation successful
0810 19:03:05.328797 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":137,"threads_started":1}
I20260810 19:03:05.329645 29576 leader_election.cc:302] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8f082868a6f84966b038df4cfe7e1665; no voters:
I20260810 19:03:05.329999 29576 leader_election.cc:288] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:05.330070 29586 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:05.330295 29586 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [term 1 LEADER]: Becoming Leader. State: Replica: 8f082868a6f84966b038df4cfe7e1665, State: Running, Role: LEADER
I20260810 19:03:05.330663 29586 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f082868a6f84966b038df4cfe7e1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } }
I20260810 19:03:05.331852 29586 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8f082868a6f84966b038df4cfe7e1665. Latest consensus state: current_term: 1 leader_uuid: "8f082868a6f84966b038df4cfe7e1665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f082868a6f84966b038df4cfe7e1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } } }
I20260810 19:03:05.332021 29586 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:05.332118 29576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:05.332455 29587 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8f082868a6f84966b038df4cfe7e1665" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8f082868a6f84966b038df4cfe7e1665" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41177 } } }
I20260810 19:03:05.332623 29587 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:05.332801 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:05.332954 29590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:05.337091 29590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:05.340328 29584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:05.335197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:58697 (local address 127.0.0.1:41177)
0810 19:03:05.335337 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:05.335341 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:05.335583 (+ 242us) server_negotiation.cc:407] Connection header received
0810 19:03:05.335690 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:05.335693 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:05.335744 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:05.335843 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:05.336175 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:05.336908 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:05.338260 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:05.338626 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:05.338745 (+ 119us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:05.339788 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:05.339792 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:05.339795 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:05.339808 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:05.339845 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:05.339859 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:05.339953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:05.340146 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 19:03:05.340182 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 19:03:05.344233 29590 catalog_manager.cc:1347] Generated new cluster ID: b76f5eeafb714e1ba7b7142d05251a80
I20260810 19:03:05.344310 29590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:05.355232 29590 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:05.356124 29590 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:05.364441 29590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8f082868a6f84966b038df4cfe7e1665: Generated new TSK 0
I20260810 19:03:05.365178 29590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:05.374794 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29464
W20260810 19:03:05.399665 29445 connection.cc:535] client connection to 127.0.0.1:41177 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41177): Connection reset by peer (error 104)
2026-08-10T19:03:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1633 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:05 dist-test-slave-znh6 krb5kdc[29620](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:05 dist-test-slave-znh6 krb5kdc[29620](info): set up 2 sockets
Aug 10 19:03:05 dist-test-slave-znh6 krb5kdc[29620](info): commencing operation
krb5kdc: starting...
W20260810 19:03:06.650748 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.236s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:06 dist-test-slave-znh6 krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388586, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 19:03:06.779419 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39269
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45053
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39269
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:06.891713 29643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:06.891945 29643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:06.891985 29643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:06.895869 29643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:06.895918 29643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:06.895938 29643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:06.895954 29643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:06.895969 29643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:06.900892 29643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45053
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39269
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39269
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29643
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:06.901762 29643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:06.902658 29643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:06.910923 29652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:06.910938 29656 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:06.910912 29658 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:06.911126 29643 server_base.cc:1034] running on GCE node
I20260810 19:03:06.911710 29643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:06.912731 29643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:06.913931 29643 hybrid_clock.cc:648] HybridClock initialized: now 1786388586913915 us; error 33 us; skew 500 ppm
Aug 10 19:03:06 dist-test-slave-znh6 krb5kdc[29620](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388586, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:06.916678 29643 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 19:03:06.917553 29643 webserver.cc:460] Webserver started at http://127.0.0.1:35149/ using document root <none> and password file <none>
I20260810 19:03:06.918098 29643 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:06.918195 29643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:06.918423 29643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:06.920426 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2"
format_stamp: "Formatted at 2026-08-10 19:03:06 on dist-test-slave-znh6"
server_key: "03fc22ff96d46815bf555902ff75a6dc"
server_key_iv: "8e7b56e56c6318467d67f7557ee8f3dd"
server_key_version: "encryptionkey@0"
I20260810 19:03:06.920991 29643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2"
format_stamp: "Formatted at 2026-08-10 19:03:06 on dist-test-slave-znh6"
server_key: "03fc22ff96d46815bf555902ff75a6dc"
server_key_iv: "8e7b56e56c6318467d67f7557ee8f3dd"
server_key_version: "encryptionkey@0"
I20260810 19:03:06.924599 29643 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 19:03:06.926944 29666 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:06.928071 29643 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:03:06.928212 29643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2"
format_stamp: "Formatted at 2026-08-10 19:03:06 on dist-test-slave-znh6"
server_key: "03fc22ff96d46815bf555902ff75a6dc"
server_key_iv: "8e7b56e56c6318467d67f7557ee8f3dd"
server_key_version: "encryptionkey@0"
I20260810 19:03:06.928323 29643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:06.932731 29643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:06.935828 29643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:06.936028 29643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:06.943203 29643 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39269
I20260810 19:03:06.943234 29749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39269 every 8 connection(s)
I20260810 19:03:06.944094 29643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:06.945473 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29643
I20260810 19:03:06.945623 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:06.945926 28238 external_mini_cluster.cc:1428] Setting key 29d608d5bcfe423f957f7328d55f8cf6
I20260810 19:03:06.947163 29751 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:06.951457 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:06.947363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:40137 (local address 127.0.0.1:39269)
0810 19:03:06.947781 (+ 418us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:06.947799 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:06.947839 (+ 40us) server_negotiation.cc:407] Connection header received
0810 19:03:06.948662 (+ 823us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:06.948685 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:06.949035 (+ 350us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:06.949384 (+ 349us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:06.950035 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:06.950040 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:06.950054 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:06.950068 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:06.950140 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:06.950167 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:06.950313 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:06.950381 (+ 68us) server_negotiation.cc:299] Negotiation successful
0810 19:03:06.950534 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":117,"threads_started":1}
I20260810 19:03:06.953727 29751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2: Bootstrap starting.
I20260810 19:03:06.954089 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:06.956494 29751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:06.957329 29751 log.cc:824] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:06.958115 29754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:06.956390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.140:34429 (local address 127.0.0.1:39269)
0810 19:03:06.956537 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:06.956543 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:06.956795 (+ 252us) server_negotiation.cc:407] Connection header received
0810 19:03:06.956877 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:06.956880 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:06.956913 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:06.956996 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:06.957661 (+ 665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:06.957665 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:06.957671 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:06.957688 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:06.957718 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:06.957731 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:06.957850 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:06.957953 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 19:03:06.957989 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 19:03:06.959499 29751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2: No bootstrap required, opened a new log
I20260810 19:03:06.963125 29751 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260810 19:03:06.963344 29751 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:06.963382 29751 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8989d320e6a4ffa9a4d42dc898ca2f2, State: Initialized, Role: FOLLOWER
I20260810 19:03:06.963723 29751 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260810 19:03:06.963831 29751 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:06.963874 29751 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:06.963944 29751 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:06.964706 29751 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260810 19:03:06.964989 29751 leader_election.cc:302] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8989d320e6a4ffa9a4d42dc898ca2f2; no voters:
I20260810 19:03:06.965291 29751 leader_election.cc:288] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:06.965415 29764 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:06.965695 29764 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [term 1 LEADER]: Becoming Leader. State: Replica: c8989d320e6a4ffa9a4d42dc898ca2f2, State: Running, Role: LEADER
I20260810 19:03:06.966043 29764 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } }
I20260810 19:03:06.966711 29751 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:06.967883 29767 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8989d320e6a4ffa9a4d42dc898ca2f2. Latest consensus state: current_term: 1 leader_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } } }
I20260810 19:03:06.968025 29767 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:06.968314 29766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8989d320e6a4ffa9a4d42dc898ca2f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39269 } } }
I20260810 19:03:06.968402 29766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:06.968444 29779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:06.971572 29779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:06.977595 29779 catalog_manager.cc:1347] Generated new cluster ID: b65902acd0334f4f8e4e5e291aae3293
I20260810 19:03:06.977707 29779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:06.985471 29779 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:06.985764 29779 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:06.996397 29779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8989d320e6a4ffa9a4d42dc898ca2f2: Generated new TSK 0
I20260810 19:03:06.997032 29779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:07.001688 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29643
2026-08-10T19:03:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1613 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:07 dist-test-slave-znh6 krb5kdc[29794](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:07 dist-test-slave-znh6 krb5kdc[29794](info): set up 2 sockets
Aug 10 19:03:07 dist-test-slave-znh6 krb5kdc[29794](info): commencing operation
krb5kdc: starting...
W20260810 19:03:08.253865 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.004s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:08 dist-test-slave-znh6 krb5kdc[29794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:08.381942 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37693
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:44183
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:37693
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:08.497872 29810 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:08.498121 29810 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:08.498162 29810 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:08.502084 29810 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:08.502137 29810 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:08.502156 29810 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:08.502173 29810 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:08.502187 29810 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:08.507251 29810 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:44183
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:37693
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:37693
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29810
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:08.508169 29810 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:08.509115 29810 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:08.517475 29818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:08.517477 29823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:08.517477 29819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:08.517603 29810 server_base.cc:1034] running on GCE node
I20260810 19:03:08.518313 29810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:08.519412 29810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:08.520577 29810 hybrid_clock.cc:648] HybridClock initialized: now 1786388588520550 us; error 39 us; skew 500 ppm
Aug 10 19:03:08 dist-test-slave-znh6 krb5kdc[29794](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388588, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:08.523481 29810 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:08.524328 29810 webserver.cc:460] Webserver started at http://127.27.147.190:38377/ using document root <none> and password file <none>
I20260810 19:03:08.524879 29810 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:08.524984 29810 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:08.525220 29810 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:08.527384 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "937f3045f9c346b9b7ddf7be449b490e"
format_stamp: "Formatted at 2026-08-10 19:03:08 on dist-test-slave-znh6"
server_key: "fb3328b86b989a2075e4b6f4cf3be287"
server_key_iv: "1853ef86c9a0de6d6de47bb63cdc6ff5"
server_key_version: "encryptionkey@0"
I20260810 19:03:08.528005 29810 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "937f3045f9c346b9b7ddf7be449b490e"
format_stamp: "Formatted at 2026-08-10 19:03:08 on dist-test-slave-znh6"
server_key: "fb3328b86b989a2075e4b6f4cf3be287"
server_key_iv: "1853ef86c9a0de6d6de47bb63cdc6ff5"
server_key_version: "encryptionkey@0"
I20260810 19:03:08.531658 29810 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:03:08.534214 29832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:08.535414 29810 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:08.535584 29810 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "937f3045f9c346b9b7ddf7be449b490e"
format_stamp: "Formatted at 2026-08-10 19:03:08 on dist-test-slave-znh6"
server_key: "fb3328b86b989a2075e4b6f4cf3be287"
server_key_iv: "1853ef86c9a0de6d6de47bb63cdc6ff5"
server_key_version: "encryptionkey@0"
I20260810 19:03:08.535714 29810 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:08.549530 29810 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:08.552758 29810 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:08.552971 29810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:08.560271 29810 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:37693
I20260810 19:03:08.560295 29913 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:37693 every 8 connection(s)
I20260810 19:03:08.561187 29810 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:08.564016 29914 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:08.568017 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29810
I20260810 19:03:08.568178 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:08.568476 28238 external_mini_cluster.cc:1428] Setting key d119029241b2b00a5fce9cdee511c8ad
I20260810 19:03:08.569793 29914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e: Bootstrap starting.
I20260810 19:03:08.572548 29914 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:08.573498 29914 log.cc:824] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:08.575573 29914 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e: No bootstrap required, opened a new log
Aug 10 19:03:08 dist-test-slave-znh6 krb5kdc[29794](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388588, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:03:08.579622 29914 raft_consensus.cc:348] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937f3045f9c346b9b7ddf7be449b490e" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37693 } }
I20260810 19:03:08.579859 29914 raft_consensus.cc:374] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:08.579977 29914 raft_consensus.cc:729] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 937f3045f9c346b9b7ddf7be449b490e, State: Initialized, Role: FOLLOWER
I20260810 19:03:08.580518 29914 consensus_queue.cc:260] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937f3045f9c346b9b7ddf7be449b490e" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37693 } }
I20260810 19:03:08.580678 29914 raft_consensus.cc:388] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:08.580752 29914 raft_consensus.cc:482] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:08.580855 29914 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:08.581909 29914 raft_consensus.cc:504] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937f3045f9c346b9b7ddf7be449b490e" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37693 } }
I20260810 19:03:08.582269 29914 leader_election.cc:302] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 937f3045f9c346b9b7ddf7be449b490e; no voters:
I20260810 19:03:08.582614 29914 leader_election.cc:288] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:08.582707 29921 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:08.582911 29921 raft_consensus.cc:686] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [term 1 LEADER]: Becoming Leader. State: Replica: 937f3045f9c346b9b7ddf7be449b490e, State: Running, Role: LEADER
I20260810 19:03:08.583277 29921 consensus_queue.cc:237] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937f3045f9c346b9b7ddf7be449b490e" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37693 } }
I20260810 19:03:08.583953 29914 sys_catalog.cc:564] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:08.584566 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:08.569940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44231 (local address 127.27.147.190:37693)
0810 19:03:08.570351 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:08.570370 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:08.570412 (+ 42us) server_negotiation.cc:407] Connection header received
0810 19:03:08.571288 (+ 876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:08.571312 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:08.571722 (+ 410us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:08.572083 (+ 361us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:08.572777 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:08.574207 (+ 1430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:08.575162 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:08.575833 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:08.576017 (+ 184us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:08.578918 (+ 2901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:08.578923 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:08.578936 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:08.578963 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:08.581124 (+ 2161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:08.581645 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:08.581651 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:08.581656 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:08.581812 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:08.582173 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:08.582177 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:08.582179 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:08.582638 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:08.582911 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:08.583514 (+ 603us) server_negotiation.cc:299] Negotiation successful
0810 19:03:08.583704 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":259,"thread_start_us":109,"threads_started":1}
I20260810 19:03:08.584735 29921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 937f3045f9c346b9b7ddf7be449b490e. Latest consensus state: current_term: 1 leader_uuid: "937f3045f9c346b9b7ddf7be449b490e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937f3045f9c346b9b7ddf7be449b490e" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37693 } } }
I20260810 19:03:08.584856 29921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:08.586226 29923 sys_catalog.cc:455] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "937f3045f9c346b9b7ddf7be449b490e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "937f3045f9c346b9b7ddf7be449b490e" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 37693 } } }
I20260810 19:03:08.586395 29923 sys_catalog.cc:458] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:08.586743 29928 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:08.587996 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:08.590260 29928 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:08.596328 29928 catalog_manager.cc:1347] Generated new cluster ID: 04de39d2ad3149f3ae0b430f89c1847b
I20260810 19:03:08.596391 29928 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:08.601727 29919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:08.590004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40007 (local address 127.27.147.190:37693)
0810 19:03:08.590111 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:08.590115 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:08.592589 (+ 2474us) server_negotiation.cc:407] Connection header received
0810 19:03:08.592713 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:08.592716 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:08.592772 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:08.592858 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:08.593700 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:08.594374 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:08.595819 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:08.596201 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:08.596306 (+ 105us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:08.598162 (+ 1856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:08.598165 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:08.598168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:08.598193 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:08.599885 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:08.600435 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:08.600438 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:08.600440 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:08.600519 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:08.600864 (+ 345us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:08.600866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:08.600868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:08.601048 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:08.601177 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:08.601487 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 19:03:08.601581 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 19:03:08.616775 29928 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:08.617956 29928 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:08.629098 29928 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 937f3045f9c346b9b7ddf7be449b490e: Generated new TSK 0
I20260810 19:03:08.629786 29928 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:08.643615 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29810
W20260810 19:03:08.659322 29791 connection.cc:535] client connection to 127.27.147.190:37693 recv error: Network error: failed to read from TLS socket (remote: 127.27.147.190:37693): Connection reset by peer (error 104)
2026-08-10T19:03:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1646 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:08 dist-test-slave-znh6 krb5kdc[29963](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:08 dist-test-slave-znh6 krb5kdc[29963](info): set up 2 sockets
Aug 10 19:03:08 dist-test-slave-znh6 krb5kdc[29963](info): commencing operation
krb5kdc: starting...
W20260810 19:03:09.883237 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.209s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:09 dist-test-slave-znh6 krb5kdc[29963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388589, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:10.007618 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:33631
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:33579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:33631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:10.118410 29981 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:10.118623 29981 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:10.118680 29981 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:10.122648 29981 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:10.122699 29981 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:10.122717 29981 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:10.122733 29981 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:10.122748 29981 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:10.127717 29981 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:33579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:33631
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:33631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29981
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:10.128607 29981 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:10.129545 29981 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:10.138175 29992 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:10.138182 29989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:10.138325 29981 server_base.cc:1034] running on GCE node
W20260810 19:03:10.138182 29988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:10.139067 29981 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:10.140077 29981 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:10.141244 29981 hybrid_clock.cc:648] HybridClock initialized: now 1786388590141231 us; error 23 us; skew 500 ppm
Aug 10 19:03:10 dist-test-slave-znh6 krb5kdc[29963](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388590, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:10.144042 29981 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:10.144876 29981 webserver.cc:460] Webserver started at http://127.27.147.190:36335/ using document root <none> and password file <none>
I20260810 19:03:10.145401 29981 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:10.145476 29981 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:10.145715 29981 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:10.147707 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "71146916e154450085a6ac42c3f948d6"
format_stamp: "Formatted at 2026-08-10 19:03:10 on dist-test-slave-znh6"
server_key: "5960e60e7c05143b04f1fcd652a93b7a"
server_key_iv: "b3e44ffd83fc60e1aabd8cc952f01a36"
server_key_version: "encryptionkey@0"
I20260810 19:03:10.148247 29981 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "71146916e154450085a6ac42c3f948d6"
format_stamp: "Formatted at 2026-08-10 19:03:10 on dist-test-slave-znh6"
server_key: "5960e60e7c05143b04f1fcd652a93b7a"
server_key_iv: "b3e44ffd83fc60e1aabd8cc952f01a36"
server_key_version: "encryptionkey@0"
I20260810 19:03:10.151863 29981 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:03:10.154116 30001 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:10.155311 29981 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:10.155483 29981 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "71146916e154450085a6ac42c3f948d6"
format_stamp: "Formatted at 2026-08-10 19:03:10 on dist-test-slave-znh6"
server_key: "5960e60e7c05143b04f1fcd652a93b7a"
server_key_iv: "b3e44ffd83fc60e1aabd8cc952f01a36"
server_key_version: "encryptionkey@0"
I20260810 19:03:10.155593 29981 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:10.180675 29981 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:10.183837 29981 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:10.184113 29981 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:10.191445 29981 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:33631
I20260810 19:03:10.191474 30077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:33631 every 8 connection(s)
I20260810 19:03:10.192368 29981 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:10.193657 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 29981
I20260810 19:03:10.193819 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:10.194160 28238 external_mini_cluster.cc:1428] Setting key 734acc24562f3e112edbd6fc78831150
I20260810 19:03:10.195670 30078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:10.201508 30078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6: Bootstrap starting.
I20260810 19:03:10.203274 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:10.195641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46277 (local address 127.27.147.190:33631)
0810 19:03:10.196198 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:10.196214 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:10.196252 (+ 38us) server_negotiation.cc:407] Connection header received
0810 19:03:10.196899 (+ 647us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:10.196914 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:10.197228 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:10.197582 (+ 354us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:10.197987 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:10.199054 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:10.199993 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:10.200467 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:10.200646 (+ 179us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:10.201563 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:10.201588 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:10.201607 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:10.201996 (+ 389us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:10.202048 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:10.202071 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:10.202218 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:10.202371 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 19:03:10.202514 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":256,"threads_started":1}
I20260810 19:03:10.203931 30078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:10.204809 30078 log.cc:824] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:10.206321 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:10.207022 30078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6: No bootstrap required, opened a new log
I20260810 19:03:10.211562 30078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71146916e154450085a6ac42c3f948d6" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33631 } }
I20260810 19:03:10.211784 30078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:10.211886 30078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71146916e154450085a6ac42c3f948d6, State: Initialized, Role: FOLLOWER
I20260810 19:03:10.212384 30078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71146916e154450085a6ac42c3f948d6" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33631 } }
I20260810 19:03:10.212575 30078 raft_consensus.cc:388] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:10.212662 30078 raft_consensus.cc:482] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:10.212780 30078 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:10.212807 30081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:10.208443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52381 (local address 127.27.147.190:33631)
0810 19:03:10.208567 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:10.208574 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:10.208737 (+ 163us) server_negotiation.cc:407] Connection header received
0810 19:03:10.208810 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:10.208813 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:10.208856 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:10.208952 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:10.209186 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:10.210057 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:10.210938 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:10.211280 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:10.211416 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:10.212208 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:10.212212 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:10.212214 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:10.212226 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:10.212257 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:10.212268 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:10.212339 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:10.212621 (+ 282us) server_negotiation.cc:299] Negotiation successful
0810 19:03:10.212665 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260810 19:03:10.214051 30078 raft_consensus.cc:504] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71146916e154450085a6ac42c3f948d6" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33631 } }
I20260810 19:03:10.214419 30078 leader_election.cc:302] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 71146916e154450085a6ac42c3f948d6; no voters:
I20260810 19:03:10.214740 30078 leader_election.cc:288] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:10.214875 30091 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:10.215143 30091 raft_consensus.cc:686] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [term 1 LEADER]: Becoming Leader. State: Replica: 71146916e154450085a6ac42c3f948d6, State: Running, Role: LEADER
I20260810 19:03:10.215570 30091 consensus_queue.cc:237] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71146916e154450085a6ac42c3f948d6" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33631 } }
I20260810 19:03:10.216001 30078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:10.217406 30095 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "71146916e154450085a6ac42c3f948d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71146916e154450085a6ac42c3f948d6" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33631 } } }
I20260810 19:03:10.217376 30097 sys_catalog.cc:455] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 71146916e154450085a6ac42c3f948d6. Latest consensus state: current_term: 1 leader_uuid: "71146916e154450085a6ac42c3f948d6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "71146916e154450085a6ac42c3f948d6" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 33631 } } }
I20260810 19:03:10.217579 30095 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:10.217644 30097 sys_catalog.cc:458] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:10.218621 30108 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:10.221200 30108 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:10.226468 30108 catalog_manager.cc:1347] Generated new cluster ID: 81d03e24f78b4f0eb5f037a217a83d5e
I20260810 19:03:10.226567 30108 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:10.251631 30108 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:10.252558 30108 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:10.261299 30108 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 71146916e154450085a6ac42c3f948d6: Generated new TSK 0
I20260810 19:03:10.261952 30108 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:10.269603 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 29981
W20260810 19:03:10.285367 29957 connection.cc:535] client connection to 127.27.147.190:33631 recv error: Network error: failed to read from TLS socket (remote: 127.27.147.190:33631): Connection reset by peer (error 104)
2026-08-10T19:03:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1626 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:10 dist-test-slave-znh6 krb5kdc[30120](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:10 dist-test-slave-znh6 krb5kdc[30120](info): set up 2 sockets
Aug 10 19:03:10 dist-test-slave-znh6 krb5kdc[30120](info): commencing operation
krb5kdc: starting...
W20260810 19:03:11.526435 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:11 dist-test-slave-znh6 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388591, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:11.669354 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:40585
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:40827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:40585
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:11.778745 30139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:11.778992 30139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:11.779031 30139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:11.782907 30139 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:11.782953 30139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:11.782971 30139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:11.783037 30139 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:11.783073 30139 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:11.788048 30139 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:40827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:40585
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:40585
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:11.788975 30139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:11.789918 30139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:11.798326 30148 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:11.798328 30151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:11.798341 30145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:11.798449 30139 server_base.cc:1034] running on GCE node
I20260810 19:03:11.799186 30139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:11.800279 30139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:11.801481 30139 hybrid_clock.cc:648] HybridClock initialized: now 1786388591801455 us; error 41 us; skew 500 ppm
Aug 10 19:03:11 dist-test-slave-znh6 krb5kdc[30120](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388591, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:11.804230 30139 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:11.805138 30139 webserver.cc:460] Webserver started at http://127.27.147.190:45267/ using document root <none> and password file <none>
I20260810 19:03:11.805692 30139 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:11.805786 30139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:11.806033 30139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:11.808044 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "d79ca241a6a2476592f2aeeec4102d3d"
format_stamp: "Formatted at 2026-08-10 19:03:11 on dist-test-slave-znh6"
server_key: "3c8498c6b4b59949a24c77cb394b83e3"
server_key_iv: "17d0debdff97e99d90b7b9d19b159c8e"
server_key_version: "encryptionkey@0"
I20260810 19:03:11.808607 30139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "d79ca241a6a2476592f2aeeec4102d3d"
format_stamp: "Formatted at 2026-08-10 19:03:11 on dist-test-slave-znh6"
server_key: "3c8498c6b4b59949a24c77cb394b83e3"
server_key_iv: "17d0debdff97e99d90b7b9d19b159c8e"
server_key_version: "encryptionkey@0"
I20260810 19:03:11.812196 30139 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:03:11.814435 30158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:11.815647 30139 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:11.815796 30139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "d79ca241a6a2476592f2aeeec4102d3d"
format_stamp: "Formatted at 2026-08-10 19:03:11 on dist-test-slave-znh6"
server_key: "3c8498c6b4b59949a24c77cb394b83e3"
server_key_iv: "17d0debdff97e99d90b7b9d19b159c8e"
server_key_version: "encryptionkey@0"
I20260810 19:03:11.815908 30139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:11.820343 30139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:11.823128 30139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:11.823344 30139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:11.830571 30139 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:40585
I20260810 19:03:11.830600 30239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:40585 every 8 connection(s)
I20260810 19:03:11.831511 30139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:11.834110 30242 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:11.835084 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30139
I20260810 19:03:11.835244 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:11.835557 28238 external_mini_cluster.cc:1428] Setting key 16aeb2ec9e9fb36388665de11361a9c9
I20260810 19:03:11.840595 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:11.836870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37903 (local address 127.27.147.190:40585)
0810 19:03:11.837335 (+ 465us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:11.837351 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:11.837387 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:03:11.838021 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:11.838037 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:11.838301 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:11.838571 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:11.839435 (+ 864us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:11.839440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:11.839451 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:11.839462 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:11.839512 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:11.839530 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:11.839637 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:11.839762 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 19:03:11.839944 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":157,"threads_started":1}
I20260810 19:03:11.840986 30242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d: Bootstrap starting.
I20260810 19:03:11.843356 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:11.843519 30242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:11.844237 30242 log.cc:824] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:11.846352 30242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d: No bootstrap required, opened a new log
I20260810 19:03:11.847147 30247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:11.845490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40063 (local address 127.27.147.190:40585)
0810 19:03:11.845619 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:11.845624 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:11.845819 (+ 195us) server_negotiation.cc:407] Connection header received
0810 19:03:11.845946 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:11.845948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:11.845974 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:11.846035 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:11.846751 (+ 716us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:11.846755 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:11.846757 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:11.846769 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:11.846796 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:11.846808 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:11.846912 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:11.847011 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 19:03:11.847046 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 19:03:11.850078 30242 raft_consensus.cc:348] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79ca241a6a2476592f2aeeec4102d3d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40585 } }
I20260810 19:03:11.850277 30242 raft_consensus.cc:374] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:11.850320 30242 raft_consensus.cc:729] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d79ca241a6a2476592f2aeeec4102d3d, State: Initialized, Role: FOLLOWER
I20260810 19:03:11.850664 30242 consensus_queue.cc:260] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79ca241a6a2476592f2aeeec4102d3d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40585 } }
I20260810 19:03:11.850772 30242 raft_consensus.cc:388] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:11.850874 30242 raft_consensus.cc:482] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:11.850972 30242 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:11.851733 30242 raft_consensus.cc:504] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79ca241a6a2476592f2aeeec4102d3d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40585 } }
I20260810 19:03:11.852026 30242 leader_election.cc:302] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d79ca241a6a2476592f2aeeec4102d3d; no voters:
I20260810 19:03:11.852259 30242 leader_election.cc:288] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:11.852388 30256 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:11.852665 30256 raft_consensus.cc:686] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [term 1 LEADER]: Becoming Leader. State: Replica: d79ca241a6a2476592f2aeeec4102d3d, State: Running, Role: LEADER
I20260810 19:03:11.853009 30256 consensus_queue.cc:237] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79ca241a6a2476592f2aeeec4102d3d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40585 } }
I20260810 19:03:11.853525 30242 sys_catalog.cc:564] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:11.854678 30258 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d79ca241a6a2476592f2aeeec4102d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79ca241a6a2476592f2aeeec4102d3d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40585 } } }
I20260810 19:03:11.854835 30258 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:11.855175 30262 sys_catalog.cc:455] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d79ca241a6a2476592f2aeeec4102d3d. Latest consensus state: current_term: 1 leader_uuid: "d79ca241a6a2476592f2aeeec4102d3d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d79ca241a6a2476592f2aeeec4102d3d" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40585 } } }
I20260810 19:03:11.855248 30269 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:11.855369 30262 sys_catalog.cc:458] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:11.858590 30269 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:11.864190 30269 catalog_manager.cc:1347] Generated new cluster ID: dd253c6e668a4f298d0cbc1ec9e1f5f7
I20260810 19:03:11.864272 30269 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:11.892068 30269 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:11.892419 30269 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:11.906847 30269 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d79ca241a6a2476592f2aeeec4102d3d: Generated new TSK 0
I20260810 19:03:11.907527 30269 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:11.910293 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30139
2026-08-10T19:03:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1640 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:11 dist-test-slave-znh6 krb5kdc[30287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:11 dist-test-slave-znh6 krb5kdc[30287](info): set up 2 sockets
Aug 10 19:03:11 dist-test-slave-znh6 krb5kdc[30287](info): commencing operation
krb5kdc: starting...
W20260810 19:03:13.163926 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.224s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:13 dist-test-slave-znh6 krb5kdc[30287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:13.291525 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:39615
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:39615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:13.403945 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:13.404201 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:13.404240 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:13.408583 30306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:13.408653 30306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:13.408672 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:13.408687 30306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:13.408702 30306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:13.413869 30306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:39615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:39615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:13.414911 30306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:13.415934 30306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:03:13.424597 30306 server_base.cc:1034] running on GCE node
W20260810 19:03:13.424633 30317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:13.424572 30316 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:13.424751 30320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:13.425339 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:13.426379 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:13.427542 30306 hybrid_clock.cc:648] HybridClock initialized: now 1786388593427522 us; error 31 us; skew 500 ppm
Aug 10 19:03:13 dist-test-slave-znh6 krb5kdc[30287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388593, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:13.430310 30306 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:13.431286 30306 webserver.cc:460] Webserver started at http://127.27.147.190:44077/ using document root <none> and password file <none>
I20260810 19:03:13.431838 30306 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:13.431896 30306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:13.432139 30306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:13.434127 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "0fc206270edb4fb7b10711721f621b24"
format_stamp: "Formatted at 2026-08-10 19:03:13 on dist-test-slave-znh6"
server_key: "bd6b434489e78eaa13f095f1e1f29f3b"
server_key_iv: "843e16e8de4ff4f22e69e419561d1a02"
server_key_version: "encryptionkey@0"
I20260810 19:03:13.434793 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "0fc206270edb4fb7b10711721f621b24"
format_stamp: "Formatted at 2026-08-10 19:03:13 on dist-test-slave-znh6"
server_key: "bd6b434489e78eaa13f095f1e1f29f3b"
server_key_iv: "843e16e8de4ff4f22e69e419561d1a02"
server_key_version: "encryptionkey@0"
I20260810 19:03:13.439339 30306 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 19:03:13.441635 30328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:13.442903 30306 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:13.443041 30306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "0fc206270edb4fb7b10711721f621b24"
format_stamp: "Formatted at 2026-08-10 19:03:13 on dist-test-slave-znh6"
server_key: "bd6b434489e78eaa13f095f1e1f29f3b"
server_key_iv: "843e16e8de4ff4f22e69e419561d1a02"
server_key_version: "encryptionkey@0"
I20260810 19:03:13.443151 30306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:13.467204 30306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:13.470986 30306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:13.471177 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:13.478281 30306 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:39615
I20260810 19:03:13.478292 30404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:39615 every 8 connection(s)
I20260810 19:03:13.479218 30306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:13.481962 30406 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:13.487354 30406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24: Bootstrap starting.
I20260810 19:03:13.488075 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30306
I20260810 19:03:13.488230 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:13.488504 28238 external_mini_cluster.cc:1428] Setting key 9741696ea3cda48039dabfdbcbd8b511
I20260810 19:03:13.490152 30406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:13.491073 30406 log.cc:824] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:13.493264 30406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24: No bootstrap required, opened a new log
I20260810 19:03:13.497298 30406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc206270edb4fb7b10711721f621b24" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39615 } }
I20260810 19:03:13.497579 30406 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:13.497659 30406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fc206270edb4fb7b10711721f621b24, State: Initialized, Role: FOLLOWER
Aug 10 19:03:13 dist-test-slave-znh6 krb5kdc[30287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388593, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:03:13.498176 30406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc206270edb4fb7b10711721f621b24" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39615 } }
I20260810 19:03:13.498307 30406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:13.498452 30406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:13.498593 30406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:13.499761 30406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc206270edb4fb7b10711721f621b24" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39615 } }
I20260810 19:03:13.500114 30406 leader_election.cc:302] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fc206270edb4fb7b10711721f621b24; no voters:
I20260810 19:03:13.500456 30406 leader_election.cc:288] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:13.500535 30413 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:13.500751 30413 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [term 1 LEADER]: Becoming Leader. State: Replica: 0fc206270edb4fb7b10711721f621b24, State: Running, Role: LEADER
I20260810 19:03:13.501096 30413 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc206270edb4fb7b10711721f621b24" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39615 } }
I20260810 19:03:13.501748 30406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:13.502800 30415 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fc206270edb4fb7b10711721f621b24. Latest consensus state: current_term: 1 leader_uuid: "0fc206270edb4fb7b10711721f621b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc206270edb4fb7b10711721f621b24" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39615 } } }
I20260810 19:03:13.502962 30415 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:13.503304 30414 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fc206270edb4fb7b10711721f621b24" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fc206270edb4fb7b10711721f621b24" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 39615 } } }
I20260810 19:03:13.503398 30414 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:13.503937 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:13.489661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44891 (local address 127.27.147.190:39615)
0810 19:03:13.490148 (+ 487us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:13.490170 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:13.490217 (+ 47us) server_negotiation.cc:407] Connection header received
0810 19:03:13.491097 (+ 880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:13.491122 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:13.491559 (+ 437us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:13.491877 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:13.492561 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:13.494001 (+ 1440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:13.494896 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:13.495600 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:13.495786 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:13.498483 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:13.498488 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:13.498500 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:13.498526 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:13.500386 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:13.500928 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:13.500933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:13.500938 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:13.501075 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:13.501340 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:13.501345 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:13.501347 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:13.501725 (+ 378us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:13.501922 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:13.502554 (+ 632us) server_negotiation.cc:299] Negotiation successful
0810 19:03:13.502899 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":308,"thread_start_us":118,"threads_started":1}
W20260810 19:03:13.506983 30429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:03:13.507066 30429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:03:13.507084 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:13.509006 30422 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:13.509913 30422 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:13.515888 30422 catalog_manager.cc:1347] Generated new cluster ID: 1e292435616646ca802d1071f6061853
I20260810 19:03:13.515956 30422 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:13.519388 30409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:13.509264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45949 (local address 127.27.147.190:39615)
0810 19:03:13.509372 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:13.509376 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:13.511086 (+ 1710us) server_negotiation.cc:407] Connection header received
0810 19:03:13.511231 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:13.511234 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:13.511281 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:13.511368 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:13.512013 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:13.512752 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:13.514377 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:13.514661 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:13.514782 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:13.516257 (+ 1475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:13.516261 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:13.516264 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:13.516289 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:13.517702 (+ 1413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:13.518138 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:13.518142 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:13.518144 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:13.518226 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:13.518526 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:13.518529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:13.518531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:13.518701 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:13.518868 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:13.519115 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 19:03:13.519230 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260810 19:03:13.544438 30422 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:13.545284 30422 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:13.550520 30422 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fc206270edb4fb7b10711721f621b24: Generated new TSK 0
I20260810 19:03:13.551209 30422 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:13.560009 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30306
2026-08-10T19:03:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1649 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:13 dist-test-slave-znh6 krb5kdc[30443](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:13 dist-test-slave-znh6 krb5kdc[30443](info): set up 2 sockets
Aug 10 19:03:13 dist-test-slave-znh6 krb5kdc[30443](info): commencing operation
krb5kdc: starting...
W20260810 19:03:14.818149 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.228s user 0.000s sys 0.007s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:14 dist-test-slave-znh6 krb5kdc[30443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388594, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:14.945780 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:40963
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:42837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:40963
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:15.055826 30460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:15.056097 30460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:15.056169 30460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:15.060240 30460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:15.060297 30460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:15.060315 30460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:15.060333 30460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:15.060348 30460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:15.065433 30460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:42837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:40963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:40963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:15.066335 30460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:15.067325 30460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:15.075618 30470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:15.075628 30474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:15.075630 30469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:15.075994 30460 server_base.cc:1034] running on GCE node
I20260810 19:03:15.076545 30460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:15.077587 30460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:15.078783 30460 hybrid_clock.cc:648] HybridClock initialized: now 1786388595078757 us; error 42 us; skew 500 ppm
Aug 10 19:03:15 dist-test-slave-znh6 krb5kdc[30443](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388595, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:15.087306 30460 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:15.088160 30460 webserver.cc:460] Webserver started at http://127.27.147.190:41883/ using document root <none> and password file <none>
I20260810 19:03:15.088726 30460 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:15.088825 30460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:15.089074 30460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:15.091228 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "69c8df623f3144afb42eb64dc284e8aa"
format_stamp: "Formatted at 2026-08-10 19:03:15 on dist-test-slave-znh6"
server_key: "9fb1040f3786bc213c9f19c168bab8a8"
server_key_iv: "db84e8034353ec8a9433fa13fbcf7cd5"
server_key_version: "encryptionkey@0"
I20260810 19:03:15.091809 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "69c8df623f3144afb42eb64dc284e8aa"
format_stamp: "Formatted at 2026-08-10 19:03:15 on dist-test-slave-znh6"
server_key: "9fb1040f3786bc213c9f19c168bab8a8"
server_key_iv: "db84e8034353ec8a9433fa13fbcf7cd5"
server_key_version: "encryptionkey@0"
I20260810 19:03:15.095388 30460 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260810 19:03:15.097718 30485 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:15.098955 30460 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:15.099097 30460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "69c8df623f3144afb42eb64dc284e8aa"
format_stamp: "Formatted at 2026-08-10 19:03:15 on dist-test-slave-znh6"
server_key: "9fb1040f3786bc213c9f19c168bab8a8"
server_key_iv: "db84e8034353ec8a9433fa13fbcf7cd5"
server_key_version: "encryptionkey@0"
I20260810 19:03:15.099211 30460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:15.126781 30460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:15.129940 30460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:15.130146 30460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:15.137118 30460 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:40963
I20260810 19:03:15.137148 30565 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:40963 every 8 connection(s)
I20260810 19:03:15.137986 30460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:15.140667 30566 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:15.141969 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30460
I20260810 19:03:15.142122 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:15.142470 28238 external_mini_cluster.cc:1428] Setting key b59b2e251dac960b16b533eb42909282
I20260810 19:03:15.146581 30566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa: Bootstrap starting.
I20260810 19:03:15.148722 30566 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:15.149447 30566 log.cc:824] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:15.151252 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:15.143925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47823 (local address 127.27.147.190:40963)
0810 19:03:15.144341 (+ 416us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:15.144355 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:15.144389 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:03:15.145031 (+ 642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:15.145047 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:15.145352 (+ 305us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:15.145680 (+ 328us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:15.146066 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:15.147100 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:15.148023 (+ 923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:15.148489 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:15.148667 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:15.149504 (+ 837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:15.149508 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:15.149520 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:15.149927 (+ 407us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:15.149980 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:15.150002 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:15.150112 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:15.150242 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 19:03:15.150450 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":282,"thread_start_us":117,"threads_started":1}
I20260810 19:03:15.151516 30566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa: No bootstrap required, opened a new log
I20260810 19:03:15.154035 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:15.155553 30566 raft_consensus.cc:348] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c8df623f3144afb42eb64dc284e8aa" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40963 } }
I20260810 19:03:15.155803 30566 raft_consensus.cc:374] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:15.155884 30566 raft_consensus.cc:729] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69c8df623f3144afb42eb64dc284e8aa, State: Initialized, Role: FOLLOWER
I20260810 19:03:15.156363 30566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c8df623f3144afb42eb64dc284e8aa" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40963 } }
I20260810 19:03:15.156564 30566 raft_consensus.cc:388] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:15.156644 30566 raft_consensus.cc:482] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:15.156755 30566 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:15.157784 30566 raft_consensus.cc:504] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c8df623f3144afb42eb64dc284e8aa" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40963 } }
I20260810 19:03:15.158149 30566 leader_election.cc:302] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69c8df623f3144afb42eb64dc284e8aa; no voters:
I20260810 19:03:15.158470 30566 leader_election.cc:288] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:15.158595 30582 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:15.158866 30582 raft_consensus.cc:686] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [term 1 LEADER]: Becoming Leader. State: Replica: 69c8df623f3144afb42eb64dc284e8aa, State: Running, Role: LEADER
I20260810 19:03:15.159154 30582 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c8df623f3144afb42eb64dc284e8aa" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40963 } }
I20260810 19:03:15.159811 30566 sys_catalog.cc:564] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:15.160537 30582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69c8df623f3144afb42eb64dc284e8aa. Latest consensus state: current_term: 1 leader_uuid: "69c8df623f3144afb42eb64dc284e8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c8df623f3144afb42eb64dc284e8aa" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40963 } } }
I20260810 19:03:15.160699 30570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:15.156104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38235 (local address 127.27.147.190:40963)
0810 19:03:15.156201 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:15.156205 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:15.156442 (+ 237us) server_negotiation.cc:407] Connection header received
0810 19:03:15.156522 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:15.156525 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:15.156567 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:15.156650 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:15.156992 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:15.157716 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:15.158667 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:15.159025 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:15.159123 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:15.160027 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:15.160030 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:15.160033 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:15.160044 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:15.160070 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:15.160083 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:15.160165 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:15.160504 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 19:03:15.160564 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 19:03:15.160723 30582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:15.161391 30583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69c8df623f3144afb42eb64dc284e8aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69c8df623f3144afb42eb64dc284e8aa" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 40963 } } }
I20260810 19:03:15.161535 30583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:15.161569 30594 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:15.164695 30594 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:15.170804 30594 catalog_manager.cc:1347] Generated new cluster ID: fcad266bc6e8417e8faac5c48ae4ca53
I20260810 19:03:15.170958 30594 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:15.184937 30594 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:15.186060 30594 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:15.192149 30594 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 69c8df623f3144afb42eb64dc284e8aa: Generated new TSK 0
I20260810 19:03:15.192742 30594 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:15.202527 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30460
2026-08-10T19:03:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1643 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:15 dist-test-slave-znh6 krb5kdc[30609](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:15 dist-test-slave-znh6 krb5kdc[30609](info): set up 2 sockets
Aug 10 19:03:15 dist-test-slave-znh6 krb5kdc[30609](info): commencing operation
krb5kdc: starting...
W20260810 19:03:16.463232 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.230s user 0.006s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:16 dist-test-slave-znh6 krb5kdc[30609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388596, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:16.589267 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34571
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:35803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:34571
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:16.698840 30631 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:16.699095 30631 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:16.699167 30631 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:16.703051 30631 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:16.703131 30631 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:16.703169 30631 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:16.703225 30631 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:16.703299 30631 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:16.708316 30631 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:35803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:34571
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30631
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:16.709224 30631 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:16.710232 30631 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:16.718472 30638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:16.718465 30641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:16.718482 30639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:16.718631 30631 server_base.cc:1034] running on GCE node
I20260810 19:03:16.719285 30631 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:16.720288 30631 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:16.721481 30631 hybrid_clock.cc:648] HybridClock initialized: now 1786388596721464 us; error 29 us; skew 500 ppm
Aug 10 19:03:16 dist-test-slave-znh6 krb5kdc[30609](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388596, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:16.724153 30631 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:16.724959 30631 webserver.cc:460] Webserver started at http://127.27.147.190:41757/ using document root <none> and password file <none>
I20260810 19:03:16.725441 30631 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:16.725499 30631 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:16.725649 30631 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:16.727541 30631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "a7b0a2118e3f4c139408ee40010e3928"
format_stamp: "Formatted at 2026-08-10 19:03:16 on dist-test-slave-znh6"
server_key: "e0e21d46d9ced1b66f667c1e18246d52"
server_key_iv: "d99a8af38813b3543a1272b5a4f6bd15"
server_key_version: "encryptionkey@0"
I20260810 19:03:16.728022 30631 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "a7b0a2118e3f4c139408ee40010e3928"
format_stamp: "Formatted at 2026-08-10 19:03:16 on dist-test-slave-znh6"
server_key: "e0e21d46d9ced1b66f667c1e18246d52"
server_key_iv: "d99a8af38813b3543a1272b5a4f6bd15"
server_key_version: "encryptionkey@0"
I20260810 19:03:16.731462 30631 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 19:03:16.733785 30654 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:16.735037 30631 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:16.735162 30631 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "a7b0a2118e3f4c139408ee40010e3928"
format_stamp: "Formatted at 2026-08-10 19:03:16 on dist-test-slave-znh6"
server_key: "e0e21d46d9ced1b66f667c1e18246d52"
server_key_iv: "d99a8af38813b3543a1272b5a4f6bd15"
server_key_version: "encryptionkey@0"
I20260810 19:03:16.735255 30631 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:16.739557 30631 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:16.742202 30631 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:16.742386 30631 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:16.749630 30631 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:34571
I20260810 19:03:16.749650 30728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:34571 every 8 connection(s)
I20260810 19:03:16.750490 30631 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:16.753258 30730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:16.754776 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30631
I20260810 19:03:16.754944 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:16.755285 28238 external_mini_cluster.cc:1428] Setting key cac8376cf3e4fb9c454c5634320e4778
I20260810 19:03:16.759498 30730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928: Bootstrap starting.
I20260810 19:03:16.760298 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:16.756692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59637 (local address 127.27.147.190:34571)
0810 19:03:16.757115 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:16.757130 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:16.757164 (+ 34us) server_negotiation.cc:407] Connection header received
0810 19:03:16.757774 (+ 610us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:16.757790 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:16.758049 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:16.758371 (+ 322us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:16.759183 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:16.759189 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:16.759200 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:16.759211 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:16.759261 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:16.759280 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:16.759438 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:16.759508 (+ 70us) server_negotiation.cc:299] Negotiation successful
0810 19:03:16.759654 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":131,"threads_started":1}
I20260810 19:03:16.761787 30730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:16.762614 30730 log.cc:824] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:16.762940 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:16.764657 30730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928: No bootstrap required, opened a new log
I20260810 19:03:16.766803 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:16.765147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41073 (local address 127.27.147.190:34571)
0810 19:03:16.765265 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:16.765269 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:16.765456 (+ 187us) server_negotiation.cc:407] Connection header received
0810 19:03:16.765583 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:16.765586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:16.765622 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:16.765708 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:16.766396 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:16.766400 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:16.766402 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 19:03:16.766414 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:16.766440 (+ 26us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 19:03:16.766458 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:16.766558 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:16.766636 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 19:03:16.766675 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:03:16.768301 30730 raft_consensus.cc:348] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b0a2118e3f4c139408ee40010e3928" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34571 } }
I20260810 19:03:16.768513 30730 raft_consensus.cc:374] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 19:03:16.768548 30730 raft_consensus.cc:729] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a7b0a2118e3f4c139408ee40010e3928, State: Initialized, Role: FOLLOWER
I20260810 19:03:16.768937 30730 consensus_queue.cc:260] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b0a2118e3f4c139408ee40010e3928" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34571 } }
I20260810 19:03:16.769057 30730 raft_consensus.cc:388] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 19:03:16.769102 30730 raft_consensus.cc:482] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 19:03:16.769172 30730 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:16.769974 30730 raft_consensus.cc:504] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b0a2118e3f4c139408ee40010e3928" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34571 } }
I20260810 19:03:16.770256 30730 leader_election.cc:302] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a7b0a2118e3f4c139408ee40010e3928; no voters:
I20260810 19:03:16.770498 30730 leader_election.cc:288] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 19:03:16.770771 30744 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:16.771037 30744 raft_consensus.cc:686] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [term 1 LEADER]: Becoming Leader. State: Replica: a7b0a2118e3f4c139408ee40010e3928, State: Running, Role: LEADER
I20260810 19:03:16.771416 30744 consensus_queue.cc:237] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b0a2118e3f4c139408ee40010e3928" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34571 } }
I20260810 19:03:16.771865 30730 sys_catalog.cc:564] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:16.772682 30744 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a7b0a2118e3f4c139408ee40010e3928. Latest consensus state: current_term: 1 leader_uuid: "a7b0a2118e3f4c139408ee40010e3928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b0a2118e3f4c139408ee40010e3928" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34571 } } }
I20260810 19:03:16.772863 30744 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:16.773322 30760 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:16.773833 30745 sys_catalog.cc:455] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a7b0a2118e3f4c139408ee40010e3928" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a7b0a2118e3f4c139408ee40010e3928" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34571 } } }
I20260810 19:03:16.773967 30745 sys_catalog.cc:458] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:16.776590 30760 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:16.782601 30760 catalog_manager.cc:1347] Generated new cluster ID: 5007699474f7422fb26e3aafb17d7393
I20260810 19:03:16.782678 30760 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:16.793998 30760 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:16.794291 30760 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:16.804046 30760 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a7b0a2118e3f4c139408ee40010e3928: Generated new TSK 0
I20260810 19:03:16.804739 30760 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 19:03:16.813681 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30631
2026-08-10T19:03:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1609 ms)
[----------] 18 tests from AuthTokenIssuingTest (29356 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:16 dist-test-slave-znh6 krb5kdc[30777](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:16 dist-test-slave-znh6 krb5kdc[30777](info): set up 2 sockets
Aug 10 19:03:16 dist-test-slave-znh6 krb5kdc[30777](info): commencing operation
krb5kdc: starting...
W20260810 19:03:18.055051 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.213s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:18.180052 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:43437
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:43437,127.27.147.189:33883,127.27.147.188:45671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:18.290100 30792 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:03:18.290341 30792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:18.290364 30792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:18.290472 30792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:18.294355 30792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:18.294409 30792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:18.294430 30792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:18.294445 30792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:18.294487 30792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:18.299394 30792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:43437,127.27.147.189:33883,127.27.147.188:45671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:43437
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:18.300310 30792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:18.301286 30792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:18.309738 30801 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:18.309749 30800 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:18.309800 30803 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:18.310024 30792 server_base.cc:1034] running on GCE node
I20260810 19:03:18.310667 30792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:18.311707 30792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:18.312870 30792 hybrid_clock.cc:648] HybridClock initialized: now 1786388598312852 us; error 29 us; skew 500 ppm
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:18.315688 30792 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:18.316491 30792 webserver.cc:460] Webserver started at http://127.27.147.190:46481/ using document root <none> and password file <none>
I20260810 19:03:18.317018 30792 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:18.317077 30792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:18.317324 30792 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:18.319757 30792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "494533e0fa3d4f4bb8b12aac5134b5f2"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "e091b2da6ace03c58d8c5e736c838407"
server_key_iv: "6eaf7460641a2555b9969fec41696028"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.320385 30792 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "494533e0fa3d4f4bb8b12aac5134b5f2"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "e091b2da6ace03c58d8c5e736c838407"
server_key_iv: "6eaf7460641a2555b9969fec41696028"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.323938 30792 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.003s
I20260810 19:03:18.326275 30811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:18.327493 30792 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260810 19:03:18.327644 30792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "494533e0fa3d4f4bb8b12aac5134b5f2"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "e091b2da6ace03c58d8c5e736c838407"
server_key_iv: "6eaf7460641a2555b9969fec41696028"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.327749 30792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:18.344180 30792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:18.347358 30792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:18.347559 30792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:18.354707 30792 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:43437
I20260810 19:03:18.355010 30888 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:43437 every 8 connection(s)
I20260810 19:03:18.355719 30792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:18.358363 30889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:18.362071 30889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:18.365836 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30792
I20260810 19:03:18.365993 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:18.366302 28238 external_mini_cluster.cc:1428] Setting key cabb98f040e429efa7a6745946a9ae2d
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.189@KRBTEST.COM; defaulting to no policy
I20260810 19:03:18.376621 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.364274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:43437 (local address 127.0.0.1:45982)
0810 19:03:18.364768 (+ 494us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.364811 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.365712 (+ 901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.366462 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.366467 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.366929 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.367157 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.367169 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.368305 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.368308 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.369022 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.369031 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.369685 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.369688 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.369807 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.370385 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.370402 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.371839 (+ 1437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.373718 (+ 1879us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.373729 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.373738 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.373963 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.374282 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.374285 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.374287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.374397 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.374992 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.374997 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.375137 (+ 140us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.375344 (+ 207us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.375728 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":299,"mutex_wait_us":105,"thread_start_us":136,"threads_started":1}
I20260810 19:03:18.376624 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.365027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45982 (local address 127.27.147.190:43437)
0810 19:03:18.365339 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.365354 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.365399 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:03:18.366019 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.366023 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.366251 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.366469 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.367307 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.368179 (+ 872us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.369144 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.369504 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.369687 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.371984 (+ 2297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.371990 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.371999 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.372018 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.373610 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.374074 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.374086 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.374087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.374181 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.374496 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.374500 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.374501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.374883 (+ 382us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.375020 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.375420 (+ 400us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.375677 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":196,"thread_start_us":73,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.189@KRBTEST.COM" created.
I20260810 19:03:18.379172 30889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:18.379868 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.379621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38112)
0810 19:03:18.379727 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.379805 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:03:18.380120 30814 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
I20260810 19:03:18.380987 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.380793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38124)
0810 19:03:18.380895 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.380937 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 19:03:18.381213 30889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:33883: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab.
Entry for principal kudu/127.27.147.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab.
I20260810 19:03:18.394345 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:33883
--webserver_interface=127.27.147.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab
--principal=kudu/127.27.147.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:43437,127.27.147.189:33883,127.27.147.188:45671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:03:18.430480 30889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } attempt: 1
I20260810 19:03:18.431594 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.431145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38132)
0810 19:03:18.431400 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.431485 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20260810 19:03:18.432539 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.432308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38140)
0810 19:03:18.432423 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.432475 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:03:18.432744 30889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:33883: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
I20260810 19:03:18.500948 30889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } attempt: 2
I20260810 19:03:18.501870 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.501552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38144)
0810 19:03:18.501724 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.501791 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260810 19:03:18.502610 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.502413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38158)
0810 19:03:18.502514 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.502559 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:03:18.502765 30889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:33883: Network error: Client connection negotiation failed: client connection to 127.27.147.189:33883: connect: Connection refused (error 111)
W20260810 19:03:18.514326 30897 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:03:18.514588 30897 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:18.514612 30897 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:18.514653 30897 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:18.518712 30897 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:18.518766 30897 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:18.518786 30897 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:18.518802 30897 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:18.518870 30897 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:18.524376 30897 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:43437,127.27.147.189:33883,127.27.147.188:45671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab
--principal=kudu/127.27.147.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:33883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.147.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30897
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:18.525269 30897 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:18.526178 30897 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:18.534498 30904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:18.534508 30903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:18.534600 30897 server_base.cc:1034] running on GCE node
W20260810 19:03:18.534508 30907 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:18.535367 30897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:18.536360 30897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:18.537554 30897 hybrid_clock.cc:648] HybridClock initialized: now 1786388598537524 us; error 42 us; skew 500 ppm
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:18.540381 30897 init.cc:377] Logged in from keytab as kudu/127.27.147.189@KRBTEST.COM (short username kudu)
I20260810 19:03:18.541225 30897 webserver.cc:460] Webserver started at http://127.27.147.189:45159/ using document root <none> and password file <none>
I20260810 19:03:18.541805 30897 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:18.541895 30897 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:18.542124 30897 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:18.544185 30897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data/instance:
uuid: "c7958ddcd704403d8487f933c86954f9"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "1f65d80d6c89bc34eda94c7d71f081d3"
server_key_iv: "d0e2582f2e7e4e8c79d4d562c9dfe81c"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.544731 30897 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/wal/instance:
uuid: "c7958ddcd704403d8487f933c86954f9"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "1f65d80d6c89bc34eda94c7d71f081d3"
server_key_iv: "d0e2582f2e7e4e8c79d4d562c9dfe81c"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.548255 30897 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.001s
I20260810 19:03:18.550499 30916 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:18.551664 30897 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:03:18.551803 30897 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/wal
uuid: "c7958ddcd704403d8487f933c86954f9"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "1f65d80d6c89bc34eda94c7d71f081d3"
server_key_iv: "d0e2582f2e7e4e8c79d4d562c9dfe81c"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.551908 30897 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:18.571722 30897 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:18.574810 30897 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:18.575044 30897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:18.582159 30897 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.189:33883
I20260810 19:03:18.582445 30992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.189:33883 every 8 connection(s)
I20260810 19:03:18.583137 30897 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
I20260810 19:03:18.584358 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 30897
I20260810 19:03:18.584493 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-1/wal/instance
I20260810 19:03:18.584758 28238 external_mini_cluster.cc:1428] Setting key 354ff22746a3961ec78366575bdaabf9
I20260810 19:03:18.585965 30993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:18.589529 30993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.147.188@KRBTEST.COM; defaulting to no policy
I20260810 19:03:18.593976 30889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } attempt: 3
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.188@KRBTEST.COM" created.
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
I20260810 19:03:18.604990 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.591412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.27.147.190:43437)
0810 19:03:18.591547 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.591551 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.592481 (+ 930us) server_negotiation.cc:407] Connection header received
0810 19:03:18.593423 (+ 942us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.593427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.593502 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.593599 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.595012 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.595717 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.597031 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.597268 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.597390 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.600548 (+ 3158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.600555 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.600560 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.600594 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.602405 (+ 1811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.603256 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.603262 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.603267 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.603344 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.603781 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.603798 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.603800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.604039 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.604134 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.604654 (+ 520us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.604815 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":50}
I20260810 19:03:18.605607 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.594604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38172)
0810 19:03:18.594762 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.594777 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.594883 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.595412 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.595415 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.595586 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.596175 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.596186 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.597669 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.597674 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.598380 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.598390 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.599084 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.599087 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.599197 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.599646 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.599666 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.601033 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.603199 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.603204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.603206 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.603474 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.603777 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.603780 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.603784 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.603872 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.604422 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.604424 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.604517 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.604648 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.605468 (+ 820us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70,"mutex_wait_us":666}
I20260810 19:03:18.605899 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.594516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38172 (local address 127.27.147.189:33883)
0810 19:03:18.594916 (+ 400us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.594926 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.594967 (+ 41us) server_negotiation.cc:407] Connection header received
0810 19:03:18.595013 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.595018 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.595193 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.595360 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.596313 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.597523 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.598501 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.598956 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.599071 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.601191 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.601195 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.601205 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.601225 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.603088 (+ 1863us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.603601 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.603605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.603606 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.603679 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.603974 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.603977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.603979 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.604305 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.604440 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.604817 (+ 377us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.605044 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":280,"thread_start_us":143,"threads_started":1}
I20260810 19:03:18.605918 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.591853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:43437 (local address 127.0.0.1:45988)
0810 19:03:18.592335 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.592374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.593210 (+ 836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.593757 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.593763 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.594184 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.594862 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.594882 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.595869 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.595875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.596902 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.596912 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.597427 (+ 515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.597432 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.597672 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.598341 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.598368 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.600368 (+ 2000us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.602547 (+ 2179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.602561 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.602573 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.603112 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.603450 (+ 338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.603456 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.603459 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.603648 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.604207 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.604215 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.604385 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.604574 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.604936 (+ 362us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":143,"threads_started":1}
I20260810 19:03:18.607630 30993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:18.607831 30889 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:18.608500 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.608283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56044)
0810 19:03:18.608389 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.608439 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260810 19:03:18.609257 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.609012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56048)
0810 19:03:18.609152 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.609203 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 19:03:18.609403 30889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:45671: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab.
Entry for principal kudu/127.27.147.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab.
I20260810 19:03:18.613565 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:45671
--webserver_interface=127.27.147.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab
--principal=kudu/127.27.147.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:43437,127.27.147.189:33883,127.27.147.188:45671
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:03:18.615837 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.608164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38188)
0810 19:03:18.608251 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.608266 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.608340 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.608594 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.608596 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.608760 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.608874 (+ 114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.608884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.609749 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.609752 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.610318 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.610327 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.610900 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.610906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.611015 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.611398 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.611410 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.612670 (+ 1260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.614368 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.614374 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.614396 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.614648 (+ 252us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.614980 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.614983 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.614985 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.615065 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.615426 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.615430 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.615513 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.615625 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.615737 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 19:03:18.616353 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.608292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38188 (local address 127.27.147.189:33883)
0810 19:03:18.608397 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.608403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.608415 (+ 12us) server_negotiation.cc:407] Connection header received
0810 19:03:18.608441 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.608444 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.608475 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.608582 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.608995 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.609644 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.610422 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.610744 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.610906 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.612814 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.612816 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.612818 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.612861 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.614251 (+ 1390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.614743 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.614748 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.614751 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.614884 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.615160 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.615165 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.615166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.615323 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.615425 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.616031 (+ 606us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.616130 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 19:03:18.617077 30993 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:18.617800 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.617491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56064)
0810 19:03:18.617633 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.617724 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 19:03:18.618438 30923 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
I20260810 19:03:18.619388 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.619100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56080)
0810 19:03:18.619237 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.619309 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:03:18.619570 30993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:45671: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
I20260810 19:03:18.640538 30889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } attempt: 1
I20260810 19:03:18.641255 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.640992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56084)
0810 19:03:18.641130 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.641187 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 19:03:18.641979 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.641788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56098)
0810 19:03:18.641888 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.641932 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260810 19:03:18.642174 30889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:45671: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
I20260810 19:03:18.668846 30993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } attempt: 1
I20260810 19:03:18.669634 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.669322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56104)
0810 19:03:18.669483 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.669548 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 19:03:18.670409 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.670169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56108)
0810 19:03:18.670294 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.670347 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:03:18.670550 30993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:45671: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
I20260810 19:03:18.717320 30889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } attempt: 2
I20260810 19:03:18.718111 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.717832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56122)
0810 19:03:18.717978 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.718035 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 19:03:18.718987 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.718713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56124)
0810 19:03:18.718872 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.718937 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260810 19:03:18.719197 30889 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:45671: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
W20260810 19:03:18.727058 31002 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:03:18.727326 31002 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:18.727386 31002 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:18.727461 31002 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:18.731622 31002 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:18.731689 31002 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:18.731732 31002 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:18.731774 31002 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:18.731835 31002 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:18.736894 31002 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:43437,127.27.147.189:33883,127.27.147.188:45671
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab
--principal=kudu/127.27.147.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:45671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.147.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31002
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:18.737825 31002 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:18.738734 31002 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 19:03:18.738735 30993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } attempt: 2
I20260810 19:03:18.739462 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.739207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56126)
0810 19:03:18.739344 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.739396 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260810 19:03:18.740175 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.739953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56140)
0810 19:03:18.740070 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.740125 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 19:03:18.740302 30993 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:45671: Network error: Client connection negotiation failed: client connection to 127.27.147.188:45671: connect: Connection refused (error 111)
W20260810 19:03:18.747223 31016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:18.747244 31013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:18.747350 31002 server_base.cc:1034] running on GCE node
W20260810 19:03:18.747243 31011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:18.748004 31002 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:18.748936 31002 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:18.750108 31002 hybrid_clock.cc:648] HybridClock initialized: now 1786388598750064 us; error 52 us; skew 500 ppm
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:18.752767 31002 init.cc:377] Logged in from keytab as kudu/127.27.147.188@KRBTEST.COM (short username kudu)
I20260810 19:03:18.753593 31002 webserver.cc:460] Webserver started at http://127.27.147.188:44201/ using document root <none> and password file <none>
I20260810 19:03:18.754156 31002 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:18.754251 31002 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:18.754487 31002 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:18.756711 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data/instance:
uuid: "8513198ca1294ee280bb2bf1ae8b7557"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "75b199b0ed7806173bc626c7efba4972"
server_key_iv: "b28059bc49367692166176008907a997"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.757284 31002 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/wal/instance:
uuid: "8513198ca1294ee280bb2bf1ae8b7557"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "75b199b0ed7806173bc626c7efba4972"
server_key_iv: "b28059bc49367692166176008907a997"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.760978 31002 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.005s sys 0.000s
I20260810 19:03:18.763410 31028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:18.764547 31002 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260810 19:03:18.764676 31002 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/wal
uuid: "8513198ca1294ee280bb2bf1ae8b7557"
format_stamp: "Formatted at 2026-08-10 19:03:18 on dist-test-slave-znh6"
server_key: "75b199b0ed7806173bc626c7efba4972"
server_key_iv: "b28059bc49367692166176008907a997"
server_key_version: "encryptionkey@0"
I20260810 19:03:18.764780 31002 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:18.794600 31002 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:18.797339 31002 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:18.797551 31002 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:18.804502 31002 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.188:45671
I20260810 19:03:18.804790 31112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.188:45671 every 8 connection(s)
I20260810 19:03:18.805429 31002 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
I20260810 19:03:18.808128 31113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:18.810201 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31002
I20260810 19:03:18.810339 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786388305279399-28238-0/minicluster-data/master-2/wal/instance
I20260810 19:03:18.810616 28238 external_mini_cluster.cc:1428] Setting key 5f9bb39ac7522c3d11ec0cedc5906358
I20260810 19:03:18.812287 31113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:18.818356 30889 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } attempt: 3
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:03:18.825526 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.814176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46000 (local address 127.27.147.190:43437)
0810 19:03:18.814307 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.814313 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.814736 (+ 423us) server_negotiation.cc:407] Connection header received
0810 19:03:18.814945 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.814948 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.815012 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.815077 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.815824 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.816408 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.817405 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.817629 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.817740 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.821004 (+ 3264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.821008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.821011 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.821036 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.822943 (+ 1907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.823519 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.823526 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.823529 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.823624 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.824007 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.824011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.824013 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.824194 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.824311 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.824902 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.825027 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:03:18.826267 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.814087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:43437 (local address 127.0.0.1:46000)
0810 19:03:18.814572 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.814628 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.814757 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.815231 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.815235 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.815532 (+ 297us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.815683 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.815692 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.816532 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.816537 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.817280 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.817290 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.817786 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.817791 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.817937 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.818430 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.818445 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.820841 (+ 2396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.823054 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.823065 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.823074 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.823410 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.823722 (+ 312us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.823724 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.823725 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.823898 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.824312 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.824319 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.824619 (+ 300us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.824828 (+ 209us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.825142 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"mutex_wait_us":42,"thread_start_us":149,"threads_started":1}
I20260810 19:03:18.826267 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.812006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56156 (local address 127.27.147.188:45671)
0810 19:03:18.812557 (+ 551us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.812608 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.812654 (+ 46us) server_negotiation.cc:407] Connection header received
0810 19:03:18.813392 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.813409 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.813758 (+ 349us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.814095 (+ 337us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.815063 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.816106 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.816962 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.817436 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.817611 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.820948 (+ 3337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.820954 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.820966 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.820990 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.822993 (+ 2003us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.823761 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.823767 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.823769 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.823906 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.824595 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.824599 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.824600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.824901 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.825640 (+ 739us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.825746 (+ 106us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.825885 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":351,"spinlock_wait_cycles":768,"thread_start_us":123,"threads_started":1}
I20260810 19:03:18.827551 31113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:18.828747 30892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.818794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56172)
0810 19:03:18.818983 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.818997 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.819073 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.819525 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.819528 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.819729 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.819823 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.819830 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.820637 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.820644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.821332 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.821342 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.822047 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.822054 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.822158 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.822591 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.822610 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.824144 (+ 1534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.827155 (+ 3011us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.827159 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.827161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.827404 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.827715 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.827718 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.827720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.827818 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.828253 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.828257 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.828350 (+ 93us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.828436 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.828563 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125}
I20260810 19:03:18.829589 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.818903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56172 (local address 127.27.147.188:45671)
0810 19:03:18.819310 (+ 407us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.819326 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.819337 (+ 11us) server_negotiation.cc:407] Connection header received
0810 19:03:18.819363 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.819365 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.819394 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.819460 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.819961 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.820531 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.821492 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.821866 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.822359 (+ 493us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.824272 (+ 1913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.824276 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.824278 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.824301 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.827027 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.827525 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.827528 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.827529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.827612 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.827941 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.827944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.827946 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.828135 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.829209 (+ 1074us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.829284 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.829394 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":108,"threads_started":1}
I20260810 19:03:18.831450 30993 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } attempt: 3
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
I20260810 19:03:18.833078 30889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2: Bootstrap starting.
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:03:18.836268 30889 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:18.836843 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.828120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38192 (local address 127.27.147.189:33883)
0810 19:03:18.828258 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.828262 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.828284 (+ 22us) server_negotiation.cc:407] Connection header received
0810 19:03:18.828327 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.828330 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.828382 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.828467 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.829070 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.829559 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.830431 (+ 872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.830730 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.830887 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.833042 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.833046 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.833048 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.833073 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.834927 (+ 1854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.835440 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.835444 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.835446 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.835526 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.835882 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.835887 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.835889 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.836131 (+ 242us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.836202 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.836544 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.836670 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:03:18.837349 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.828029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:33883 (local address 127.0.0.1:38192)
0810 19:03:18.828116 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.828132 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.828198 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.828484 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.828488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.828653 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.828905 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.828916 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.829676 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.829683 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.830285 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.830295 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.830871 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.830875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.830987 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.831463 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.831477 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.832897 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.835034 (+ 2137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.835037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.835039 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.835324 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.835631 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.835634 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.835636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.835767 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.836268 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.836270 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.836349 (+ 79us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.837028 (+ 679us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.837163 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260810 19:03:18.837809 30889 log.cc:824] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:18.838145 31113 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:03:18.839432 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.828842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.27.147.190:43437)
0810 19:03:18.828944 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.828950 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.829009 (+ 59us) server_negotiation.cc:407] Connection header received
0810 19:03:18.829160 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.829165 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.829208 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.829287 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.830402 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.831101 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.832425 (+ 1324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.832732 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.832827 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.835621 (+ 2794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.835624 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.835627 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.835651 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.837248 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.837759 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.837763 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.837765 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.837857 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.838177 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.838181 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.838182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.838385 (+ 203us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.838563 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.839185 (+ 622us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.839290 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260810 19:03:18.840066 30889 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2: No bootstrap required, opened a new log
I20260810 19:03:18.843662 30997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.831836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56174)
0810 19:03:18.832098 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.832113 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.832184 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.832748 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.832751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.832935 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.833043 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.833049 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.834974 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.834979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.835809 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.835821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.836952 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.836955 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.837043 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.837520 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.837536 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.839170 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.841654 (+ 2484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.841657 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.841659 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.841934 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.842243 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.842250 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.842255 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.842350 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.843129 (+ 779us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.843132 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.843232 (+ 100us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.843337 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.843486 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:03:18.844261 31119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.832365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56174 (local address 127.27.147.188:45671)
0810 19:03:18.832467 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.832471 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.832483 (+ 12us) server_negotiation.cc:407] Connection header received
0810 19:03:18.832515 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.832517 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.832558 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.833146 (+ 588us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.833715 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.834795 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.836204 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.836809 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.839578 (+ 2769us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.839680 (+ 102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.839683 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.839686 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.839702 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.841521 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.842046 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.842050 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.842051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.842130 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.842462 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.842465 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.842467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.843001 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.843938 (+ 937us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.844016 (+ 78us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.844133 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 19:03:18.844411 30889 raft_consensus.cc:348] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
I20260810 19:03:18.844625 30889 raft_consensus.cc:729] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 494533e0fa3d4f4bb8b12aac5134b5f2, State: Initialized, Role: FOLLOWER
I20260810 19:03:18.844976 30889 consensus_queue.cc:260] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
I20260810 19:03:18.846462 30889 sys_catalog.cc:564] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 19:03:18 dist-test-slave-znh6 krb5kdc[30777](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388598, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
I20260810 19:03:18.847791 31117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.838558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:45671 (local address 127.0.0.1:56180)
0810 19:03:18.838740 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:18.838765 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:18.838870 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:18.839478 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:18.839480 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:18.839639 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:18.839713 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.839719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.840802 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.840807 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.841725 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:18.841739 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.842415 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.842422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:18.842511 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.842988 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:18.843001 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:18.844370 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:18.846336 (+ 1966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.846339 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.846340 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.846550 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.846795 (+ 245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:18.846797 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:18.846799 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:18.846904 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:18.847334 (+ 430us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:18.847336 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:18.847413 (+ 77us) client_negotiation.cc:769] Sending connection context
0810 19:03:18.847501 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 19:03:18.847615 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260810 19:03:18.848336 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.838800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56180 (local address 127.27.147.188:45671)
0810 19:03:18.839314 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.839317 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.839326 (+ 9us) server_negotiation.cc:407] Connection header received
0810 19:03:18.839351 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.839353 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.839381 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.839467 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.839844 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.840658 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.841850 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.842215 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.842320 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.844496 (+ 2176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.844502 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.844504 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.844527 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.846212 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.846639 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.846641 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.846643 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.846708 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.847003 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.847006 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.847008 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.847190 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.847298 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.848069 (+ 771us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.848173 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442}
I20260810 19:03:18.848747 30993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: Bootstrap starting.
W20260810 19:03:18.849481 30818 tablet.cc:2369] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 19:03:18.851233 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.840693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38196 (local address 127.27.147.189:33883)
0810 19:03:18.840781 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.840785 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.840988 (+ 203us) server_negotiation.cc:407] Connection header received
0810 19:03:18.841160 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.841163 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.841212 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.841259 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.842242 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.842978 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.844074 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.844392 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.844473 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.847384 (+ 2911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.847387 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.847390 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.847414 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.849270 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.849770 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.849773 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.849777 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.849864 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.850202 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.850205 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.850207 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.850437 (+ 230us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.850573 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.851005 (+ 432us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.851106 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 19:03:18.852759 31138 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:03:18.852838 31138 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:03:18.853099 31122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.853227 31122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:18.853614 30993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:18.853924 31113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: Bootstrap starting.
I20260810 19:03:18.854449 30993 log.cc:824] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:18.856297 31113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:18.853199 30792 master_runner.cc:186] Error getting master registration for 127.27.147.189:33883: OK, instance_id { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" instance_seqno: 1786388598580034 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:03:18.856505 30792 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:03:18.856683 30993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: No bootstrap required, opened a new log
I20260810 19:03:18.857079 31113 log.cc:824] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:18.857884 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:18.859253 30851 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46008
I20260810 19:03:18.859529 30851 raft_consensus.cc:482] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 19:03:18.859645 30851 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:18.860167 30993 raft_consensus.cc:348] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
I20260810 19:03:18.860389 30993 raft_consensus.cc:729] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7958ddcd704403d8487f933c86954f9, State: Initialized, Role: FOLLOWER
I20260810 19:03:18.860752 30993 consensus_queue.cc:260] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
I20260810 19:03:18.860955 30851 raft_consensus.cc:504] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
I20260810 19:03:18.861296 31146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.861481 31146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:18.861848 30851 leader_election.cc:288] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [CANDIDATE]: Term 1 election: Requested vote from peers c7958ddcd704403d8487f933c86954f9 (127.27.147.189:33883), 8513198ca1294ee280bb2bf1ae8b7557 (127.27.147.188:45671)
I20260810 19:03:18.862195 30993 sys_catalog.cc:564] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:18.862411 30955 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c7958ddcd704403d8487f933c86954f9"
I20260810 19:03:18.862615 30955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:18.864029 30955 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494533e0fa3d4f4bb8b12aac5134b5f2 in term 1.
I20260810 19:03:18.864765 30814 leader_election.cc:302] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 494533e0fa3d4f4bb8b12aac5134b5f2, c7958ddcd704403d8487f933c86954f9; no voters:
I20260810 19:03:18.865057 31122 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:18.865259 31122 raft_consensus.cc:686] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [term 1 LEADER]: Becoming Leader. State: Replica: 494533e0fa3d4f4bb8b12aac5134b5f2, State: Running, Role: LEADER
I20260810 19:03:18.865573 31122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
W20260810 19:03:18.867430 31169 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:03:18.867506 31169 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:03:18.867781 31170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494533e0fa3d4f4bb8b12aac5134b5f2. Latest consensus state: current_term: 1 leader_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.867926 31170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:18.872853 30897 master_runner.cc:186] Error getting master registration for 127.27.147.188:45671: OK, instance_id { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" instance_seqno: 1786388598802503 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:03:18.876969 30897 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:03:18.878969 31113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: No bootstrap required, opened a new log
I20260810 19:03:18.879019 31116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.868189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56196 (local address 127.27.147.188:45671)
0810 19:03:18.868323 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.868327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.869289 (+ 962us) server_negotiation.cc:407] Connection header received
0810 19:03:18.869473 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.869477 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.869521 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.869571 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.870677 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.871452 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.872552 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.872877 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.872956 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.874798 (+ 1842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.874802 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.874805 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.874849 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.876831 (+ 1982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.877620 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.877626 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.877631 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.877714 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.878040 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.878043 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.878044 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.878253 (+ 209us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.878356 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.878729 (+ 373us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.878876 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 19:03:18.880518 30998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.867397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38200 (local address 127.27.147.189:33883)
0810 19:03:18.867520 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.867524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.868649 (+ 1125us) server_negotiation.cc:407] Connection header received
0810 19:03:18.868969 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.868973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.869020 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.869072 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.870838 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.871511 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.873277 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.873619 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.873698 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.876228 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.876232 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.876234 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.876270 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.877929 (+ 1659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.878710 (+ 781us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.878714 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.878716 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.878804 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.879364 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.879367 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.879369 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.879582 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.879678 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.880266 (+ 588us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.880380 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:03:18.883091 31176 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 19:03:18.883597 31113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
I20260810 19:03:18.883886 31113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8513198ca1294ee280bb2bf1ae8b7557, State: Initialized, Role: FOLLOWER
I20260810 19:03:18.884166 31176 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 19:03:18.884275 31113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } }
I20260810 19:03:18.884896 31180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.885156 31180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:18.885738 31113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:18.886057 31065 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8513198ca1294ee280bb2bf1ae8b7557"
I20260810 19:03:18.886300 31065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:18.887673 31065 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 494533e0fa3d4f4bb8b12aac5134b5f2 in term 1.
I20260810 19:03:18.889093 30893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:18.868572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46016 (local address 127.27.147.190:43437)
0810 19:03:18.868671 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.868676 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.880278 (+ 11602us) server_negotiation.cc:407] Connection header received
0810 19:03:18.880444 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.880449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.880493 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.880541 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.881325 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.882073 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.883164 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:18.883513 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:18.883594 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:18.885267 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:18.885271 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:18.885274 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:18.885300 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:18.887107 (+ 1807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.887637 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.887640 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.887659 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.887787 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:18.888133 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:18.888136 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:18.888139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:18.888356 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:18.888464 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:18.888839 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 19:03:18.888952 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260810 19:03:18.892987 31065 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [term 1 FOLLOWER]: Refusing update from remote peer 494533e0fa3d4f4bb8b12aac5134b5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:03:18.893656 31170 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 19:03:18.901171 31203 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:03:18.901252 31203 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:03:18.901361 31180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494533e0fa3d4f4bb8b12aac5134b5f2. Latest consensus state: current_term: 1 leader_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.901454 31180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:18.902359 30955 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [term 1 FOLLOWER]: Refusing update from remote peer 494533e0fa3d4f4bb8b12aac5134b5f2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:03:18.902874 31170 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:03:18.906137 31146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 494533e0fa3d4f4bb8b12aac5134b5f2. Latest consensus state: current_term: 1 leader_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.906241 31146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:18.909601 31170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.909708 31170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:18.909941 31180 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.910049 31180 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:18.910315 31187 mvcc.cc:205] Tried to move back new op lower bound from 7317047701059895296 to 7317047700956725248. Current Snapshot: MvccSnapshot[applied={T|T < 7317047701059895296}]
I20260810 19:03:18.910894 31204 mvcc.cc:205] Tried to move back new op lower bound from 7317047701059895296 to 7317047700956725248. Current Snapshot: MvccSnapshot[applied={T|T < 7317047701059895296}]
I20260810 19:03:18.913940 31122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.914040 31122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:18.915056 31146 sys_catalog.cc:455] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "494533e0fa3d4f4bb8b12aac5134b5f2" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 43437 } } peers { permanent_uuid: "c7958ddcd704403d8487f933c86954f9" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 33883 } } peers { permanent_uuid: "8513198ca1294ee280bb2bf1ae8b7557" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 45671 } } }
I20260810 19:03:18.915167 31146 sys_catalog.cc:458] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:18.915584 31207 mvcc.cc:205] Tried to move back new op lower bound from 7317047701059895296 to 7317047700956725248. Current Snapshot: MvccSnapshot[applied={T|T < 7317047701059895296}]
I20260810 19:03:18.915642 31176 catalog_manager.cc:1347] Generated new cluster ID: 0376744507964fbd89d8896988225e45
I20260810 19:03:18.915691 31176 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:18.941732 31176 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:18.942961 31176 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:18.958006 31176 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 494533e0fa3d4f4bb8b12aac5134b5f2: Generated new TSK 0
I20260810 19:03:18.958892 31176 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:03:18.995986 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:18.993188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38202 (local address 127.27.147.189:33883)
0810 19:03:18.993309 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.993314 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.994424 (+ 1110us) server_negotiation.cc:407] Connection header received
0810 19:03:18.995084 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.995088 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.995159 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.995214 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.995850 (+ 636us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38202: BlockingRecv error: recv got EOF from 127.0.0.1:38202 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260810 19:03:18.997913 31116 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:18.993813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56198 (local address 127.27.147.188:45671)
0810 19:03:18.993938 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.993942 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.996730 (+ 2788us) server_negotiation.cc:407] Connection header received
0810 19:03:18.997205 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.997208 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.997256 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.997306 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.997786 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56198: BlockingRecv error: recv got EOF from 127.0.0.1:56198 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 19:03:18.999053 30893 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:18.994868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46018 (local address 127.27.147.190:43437)
0810 19:03:18.994986 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:18.994990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:18.997127 (+ 2137us) server_negotiation.cc:407] Connection header received
0810 19:03:18.998350 (+ 1223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:18.998354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:18.998400 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:18.998449 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:18.998942 (+ 493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46018: BlockingRecv error: recv got EOF from 127.0.0.1:46018 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 19:03:19.002965 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.001448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38216 (local address 127.27.147.189:33883)
0810 19:03:19.001548 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.001555 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.002043 (+ 488us) server_negotiation.cc:407] Connection header received
0810 19:03:19.002250 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.002253 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.002322 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.002371 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.002853 (+ 482us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38216: BlockingRecv error: recv got EOF from 127.0.0.1:38216 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
W20260810 19:03:19.008041 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.006580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.27.147.189:33883)
0810 19:03:19.006671 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.006675 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.007207 (+ 532us) server_negotiation.cc:407] Connection header received
0810 19:03:19.007383 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.007388 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.007434 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.007483 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.007955 (+ 472us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38230: BlockingRecv error: recv got EOF from 127.0.0.1:38230 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 19:03:19.013372 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.012006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38236 (local address 127.27.147.189:33883)
0810 19:03:19.012097 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.012100 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.012592 (+ 492us) server_negotiation.cc:407] Connection header received
0810 19:03:19.012757 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.012760 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.012801 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.012845 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.013283 (+ 438us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38236: BlockingRecv error: recv got EOF from 127.0.0.1:38236 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
W20260810 19:03:19.020931 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.019452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38246 (local address 127.27.147.189:33883)
0810 19:03:19.019573 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.019577 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.020096 (+ 519us) server_negotiation.cc:407] Connection header received
0810 19:03:19.020273 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.020276 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.020320 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.020366 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.020829 (+ 463us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38246: BlockingRecv error: recv got EOF from 127.0.0.1:38246 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 19:03:19.032662 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.031048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38254 (local address 127.27.147.189:33883)
0810 19:03:19.031143 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.031147 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.031680 (+ 533us) server_negotiation.cc:407] Connection header received
0810 19:03:19.031962 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.031965 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.032016 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.032068 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.032573 (+ 505us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38254: BlockingRecv error: recv got EOF from 127.0.0.1:38254 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20260810 19:03:19.053305 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.051555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.27.147.189:33883)
0810 19:03:19.051659 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.051663 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.052322 (+ 659us) server_negotiation.cc:407] Connection header received
0810 19:03:19.052524 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.052528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.052576 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.052642 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.053193 (+ 551us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38258: BlockingRecv error: recv got EOF from 127.0.0.1:38258 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260810 19:03:19.089555 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.088155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38270 (local address 127.27.147.189:33883)
0810 19:03:19.088278 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.088283 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.088594 (+ 311us) server_negotiation.cc:407] Connection header received
0810 19:03:19.088712 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.088716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.088792 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.088906 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.089426 (+ 520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38270: BlockingRecv error: recv got EOF from 127.0.0.1:38270 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 19:03:19.157100 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.155736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38274 (local address 127.27.147.189:33883)
0810 19:03:19.155905 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.155912 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.156169 (+ 257us) server_negotiation.cc:407] Connection header received
0810 19:03:19.156372 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.156375 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.156425 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.156526 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.156994 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38274: BlockingRecv error: recv got EOF from 127.0.0.1:38274 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
W20260810 19:03:19.288905 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.287568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38282 (local address 127.27.147.189:33883)
0810 19:03:19.287803 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.287809 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.287959 (+ 150us) server_negotiation.cc:407] Connection header received
0810 19:03:19.288147 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.288151 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.288207 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.288302 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.288782 (+ 480us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38282: BlockingRecv error: recv got EOF from 127.0.0.1:38282 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260810 19:03:19.548365 30998 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:19.547222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38286 (local address 127.27.147.189:33883)
0810 19:03:19.547436 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:19.547440 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:19.547526 (+ 86us) server_negotiation.cc:407] Connection header received
0810 19:03:19.547638 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:19.547640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:19.547691 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:19.547767 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:19.548231 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38286: BlockingRecv error: recv got EOF from 127.0.0.1:38286 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
I20260810 19:03:19.869048 31169 catalog_manager.cc:1259] Loaded cluster ID: 0376744507964fbd89d8896988225e45
I20260810 19:03:19.869120 31169 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: loading cluster ID for follower catalog manager: success
I20260810 19:03:19.870878 31169 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: acquiring CA information for follower catalog manager: success
I20260810 19:03:19.872020 31169 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c7958ddcd704403d8487f933c86954f9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:03:19.902513 31203 catalog_manager.cc:1259] Loaded cluster ID: 0376744507964fbd89d8896988225e45
I20260810 19:03:19.902590 31203 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: loading cluster ID for follower catalog manager: success
I20260810 19:03:19.904242 31203 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: acquiring CA information for follower catalog manager: success
I20260810 19:03:19.905335 31203 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8513198ca1294ee280bb2bf1ae8b7557: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:03:20.067184 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:20.062760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38296 (local address 127.27.147.189:33883)
0810 19:03:20.063066 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:20.063073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:20.063097 (+ 24us) server_negotiation.cc:407] Connection header received
0810 19:03:20.063353 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:20.063356 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:20.063564 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:20.063691 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:20.063989 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.064589 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.065496 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.065737 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.065854 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:20.066198 (+ 344us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:20.066740 (+ 542us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:20.066908 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:20.066960 (+ 52us) server_negotiation.cc:299] Negotiation successful
0810 19:03:20.067018 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":87,"threads_started":1}
I20260810 19:03:20.073705 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:20.070427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56206 (local address 127.27.147.188:45671)
0810 19:03:20.070676 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:20.070680 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:20.070692 (+ 12us) server_negotiation.cc:407] Connection header received
0810 19:03:20.070772 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:20.070774 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:20.070934 (+ 160us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:20.071043 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:20.071243 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.071736 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.072504 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.072716 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.072792 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:20.073055 (+ 263us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:20.073401 (+ 346us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:20.073491 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:20.073547 (+ 56us) server_negotiation.cc:299] Negotiation successful
0810 19:03:20.073592 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":98,"threads_started":1}
I20260810 19:03:20.081013 31303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:20.076555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38310 (local address 127.27.147.189:33883)
0810 19:03:20.076666 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:20.076672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:20.077302 (+ 630us) server_negotiation.cc:407] Connection header received
0810 19:03:20.077420 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:20.077423 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:20.077471 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:20.077557 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:20.077892 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.078651 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.079685 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.079906 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.080015 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:20.080446 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:20.080538 (+ 92us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:20.080613 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:20.080806 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 19:03:20.080860 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 19:03:20.081418 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:20.076570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46028 (local address 127.27.147.190:43437)
0810 19:03:20.076826 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:20.076830 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:20.076843 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:03:20.077037 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:20.077041 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:20.077192 (+ 151us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:20.077325 (+ 133us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:20.077572 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.078314 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.079406 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.079760 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.080056 (+ 296us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:20.080289 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:20.080579 (+ 290us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:20.080701 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:20.081226 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 19:03:20.081290 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":109,"threads_started":1}
I20260810 19:03:20.081458 31311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:20.076953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56218 (local address 127.27.147.188:45671)
0810 19:03:20.077069 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:20.077073 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:20.077653 (+ 580us) server_negotiation.cc:407] Connection header received
0810 19:03:20.077767 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:20.077770 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:20.077817 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:20.077903 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:20.078250 (+ 347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.078995 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.080162 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:20.080412 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:20.080509 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:20.080860 (+ 351us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:20.080962 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:20.081067 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:20.081273 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 19:03:20.081323 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:03:20.084599 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30792
I20260810 19:03:20.101470 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 30897
I20260810 19:03:20.114075 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31002
2026-08-10T19:03:20Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (3299 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 19:03:20 dist-test-slave-znh6 krb5kdc[31331](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 19:03:20 dist-test-slave-znh6 krb5kdc[31331](info): set up 2 sockets
Aug 10 19:03:20 dist-test-slave-znh6 krb5kdc[31331](info): commencing operation
krb5kdc: starting...
W20260810 19:03:21.368208 28238 mini_kdc.cc:121] Time spent starting KDC: real 1.226s user 0.001s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/test-user.keytab.
Aug 10 19:03:21 dist-test-slave-znh6 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T19:03:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T19:03:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.27.147.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
Entry for principal kudu/127.27.147.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab.
I20260810 19:03:21.496086 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34759
--webserver_interface=127.27.147.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:34759,127.27.147.189:42945,127.27.147.188:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 19:03:21.617190 31350 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:03:21.617552 31350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:21.617607 31350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:21.617684 31350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:21.624424 31350 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:21.624508 31350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:21.624544 31350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:21.624581 31350 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:21.624615 31350 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:21.633721 31350 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:34759,127.27.147.189:42945,127.27.147.188:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.190.keytab
--principal=kudu/127.27.147.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.190:34759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.27.147.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:21.635223 31350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:21.636579 31350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:21.646612 31358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:21.646782 31357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:21.646647 31350 server_base.cc:1034] running on GCE node
W20260810 19:03:21.646612 31360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:21.647495 31350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:21.648562 31350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:21.649752 31350 hybrid_clock.cc:648] HybridClock initialized: now 1786388601649721 us; error 51 us; skew 500 ppm
Aug 10 19:03:21 dist-test-slave-znh6 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:21.653182 31350 init.cc:377] Logged in from keytab as kudu/127.27.147.190@KRBTEST.COM (short username kudu)
I20260810 19:03:21.654146 31350 webserver.cc:460] Webserver started at http://127.27.147.190:42697/ using document root <none> and password file <none>
I20260810 19:03:21.654759 31350 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:21.654884 31350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:21.655128 31350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:21.657186 31350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data/instance:
uuid: "f3e6c577f85a4defb2e04191d656af47"
format_stamp: "Formatted at 2026-08-10 19:03:21 on dist-test-slave-znh6"
server_key: "3f5fc3c47b66c5f7d9d1f081c1e92194"
server_key_iv: "3f65e609dee1e67e7bbb8fbaeb1c6189"
server_key_version: "encryptionkey@0"
I20260810 19:03:21.657768 31350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/wal/instance:
uuid: "f3e6c577f85a4defb2e04191d656af47"
format_stamp: "Formatted at 2026-08-10 19:03:21 on dist-test-slave-znh6"
server_key: "3f5fc3c47b66c5f7d9d1f081c1e92194"
server_key_iv: "3f65e609dee1e67e7bbb8fbaeb1c6189"
server_key_version: "encryptionkey@0"
I20260810 19:03:21.661616 31350 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 19:03:21.664173 31377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:21.665454 31350 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260810 19:03:21.665596 31350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/wal
uuid: "f3e6c577f85a4defb2e04191d656af47"
format_stamp: "Formatted at 2026-08-10 19:03:21 on dist-test-slave-znh6"
server_key: "3f5fc3c47b66c5f7d9d1f081c1e92194"
server_key_iv: "3f65e609dee1e67e7bbb8fbaeb1c6189"
server_key_version: "encryptionkey@0"
I20260810 19:03:21.665704 31350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:21.704928 31350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:21.708096 31350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:21.708318 31350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:21.715396 31350 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.190:34759
I20260810 19:03:21.715731 31445 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.190:34759 every 8 connection(s)
I20260810 19:03:21.716405 31350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/data/info.pb
I20260810 19:03:21.719698 31447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:21.722998 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31350
I20260810 19:03:21.723132 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-0/wal/instance
I20260810 19:03:21.723457 28238 external_mini_cluster.cc:1428] Setting key 1575e9ee514cefddf3fbdaabebc30bbe
I20260810 19:03:21.724946 31447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.27.147.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.189@KRBTEST.COM" created.
Aug 10 19:03:21 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:03:21.743636 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.727647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34759 (local address 127.0.0.1:57774)
0810 19:03:21.728293 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.728344 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:21.729582 (+ 1238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:21.730301 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:21.730307 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:21.730740 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:21.731036 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.731050 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.732222 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.732226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.732930 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.732938 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.733586 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.733590 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.733681 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.734246 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:21.734262 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:21.737548 (+ 3286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:21.739915 (+ 2367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.739925 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.739936 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.740267 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.740676 (+ 409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.740681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.740684 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.740834 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.741651 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:21.741658 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:21.741828 (+ 170us) client_negotiation.cc:769] Sending connection context
0810 19:03:21.742026 (+ 198us) client_negotiation.cc:240] Negotiation successful
0810 19:03:21.742524 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":155,"spinlock_wait_cycles":16128,"thread_start_us":172,"threads_started":1}
I20260810 19:03:21.744410 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.727647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57774 (local address 127.27.147.190:34759)
0810 19:03:21.728036 (+ 389us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:21.728056 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:21.728443 (+ 387us) server_negotiation.cc:407] Connection header received
0810 19:03:21.729922 (+ 1479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:21.729926 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:21.730108 (+ 182us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:21.730303 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:21.731187 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.732101 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.733051 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.733433 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.733601 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.737705 (+ 4104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:21.737710 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:21.737720 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:21.737740 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:21.739755 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.740397 (+ 642us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.740406 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.740410 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.740562 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.740949 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.740956 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.740960 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.741514 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:21.741666 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:21.742133 (+ 467us) server_negotiation.cc:299] Negotiation successful
0810 19:03:21.742407 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":256,"thread_start_us":138,"threads_started":1}
I20260810 19:03:21.746901 31447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:21.747695 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.747360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33154)
0810 19:03:21.747497 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.747618 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 19:03:21.747916 31378 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
I20260810 19:03:21.748901 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.748647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33166)
0810 19:03:21.748777 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.748832 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:03:21.749054 31447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:42945: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab.
Entry for principal kudu/127.27.147.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab.
I20260810 19:03:21.753872 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:42945
--webserver_interface=127.27.147.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab
--principal=kudu/127.27.147.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:34759,127.27.147.189:42945,127.27.147.188:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:03:21.798314 31447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } attempt: 1
I20260810 19:03:21.799302 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.799002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33178)
0810 19:03:21.799159 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.799221 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 19:03:21.800050 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.799851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33184)
0810 19:03:21.799952 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.799995 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:03:21.800199 31447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:42945: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
W20260810 19:03:21.867348 31457 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:03:21.867607 31457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:21.867667 31457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:21.867759 31457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 19:03:21.868379 31447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } attempt: 2
I20260810 19:03:21.869268 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.868939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33186)
0810 19:03:21.869122 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.869188 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 19:03:21.870098 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.869896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33188)
0810 19:03:21.869998 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.870043 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260810 19:03:21.870242 31447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.189:42945: Network error: Client connection negotiation failed: client connection to 127.27.147.189:42945: connect: Connection refused (error 111)
W20260810 19:03:21.872573 31457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:21.872650 31457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:21.872700 31457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:21.872735 31457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:21.872769 31457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:21.877978 31457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:34759,127.27.147.189:42945,127.27.147.188:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.189.keytab
--principal=kudu/127.27.147.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.189:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.27.147.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:21.879088 31457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:21.880090 31457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:21.888559 31472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:21.888610 31457 server_base.cc:1034] running on GCE node
W20260810 19:03:21.888559 31467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:21.888567 31469 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:21.889318 31457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:21.890466 31457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:21.891662 31457 hybrid_clock.cc:648] HybridClock initialized: now 1786388601891633 us; error 39 us; skew 500 ppm
Aug 10 19:03:21 dist-test-slave-znh6 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:21.894310 31457 init.cc:377] Logged in from keytab as kudu/127.27.147.189@KRBTEST.COM (short username kudu)
I20260810 19:03:21.895222 31457 webserver.cc:460] Webserver started at http://127.27.147.189:33635/ using document root <none> and password file <none>
I20260810 19:03:21.895778 31457 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:21.895872 31457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:21.896106 31457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:21.898100 31457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data/instance:
uuid: "62bddad7833a4b23a66363b0f42048d5"
format_stamp: "Formatted at 2026-08-10 19:03:21 on dist-test-slave-znh6"
server_key: "f2acbd8e839c50e2279c77cb912e6967"
server_key_iv: "ca71423d8f58e249d3c1e48f209db3dd"
server_key_version: "encryptionkey@0"
I20260810 19:03:21.898674 31457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/wal/instance:
uuid: "62bddad7833a4b23a66363b0f42048d5"
format_stamp: "Formatted at 2026-08-10 19:03:21 on dist-test-slave-znh6"
server_key: "f2acbd8e839c50e2279c77cb912e6967"
server_key_iv: "ca71423d8f58e249d3c1e48f209db3dd"
server_key_version: "encryptionkey@0"
I20260810 19:03:21.902444 31457 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.001s
I20260810 19:03:21.905007 31480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:21.906181 31457 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:03:21.906327 31457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/wal
uuid: "62bddad7833a4b23a66363b0f42048d5"
format_stamp: "Formatted at 2026-08-10 19:03:21 on dist-test-slave-znh6"
server_key: "f2acbd8e839c50e2279c77cb912e6967"
server_key_iv: "ca71423d8f58e249d3c1e48f209db3dd"
server_key_version: "encryptionkey@0"
I20260810 19:03:21.906464 31457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:21.921938 31457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:21.925177 31457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:21.925331 31457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:21.932194 31457 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.189:42945
I20260810 19:03:21.932488 31552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.189:42945 every 8 connection(s)
I20260810 19:03:21.933166 31457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/data/info.pb
I20260810 19:03:21.935940 31553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:21.939615 31553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:21.941017 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31457
I20260810 19:03:21.941150 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-1/wal/instance
I20260810 19:03:21.941510 28238 external_mini_cluster.cc:1428] Setting key d88697a4a9b67ac80db65de1bb04434d
Aug 10 19:03:21 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.27.147.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.27.147.188@KRBTEST.COM" created.
I20260810 19:03:21.954530 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.941513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57786 (local address 127.27.147.190:34759)
0810 19:03:21.941675 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:21.941680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:21.942629 (+ 949us) server_negotiation.cc:407] Connection header received
0810 19:03:21.944190 (+ 1561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:21.944196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:21.944268 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:21.944371 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:21.945453 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.946304 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.947603 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.948040 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.948173 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.950787 (+ 2614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:21.950793 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:21.950798 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:21.950857 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:21.952382 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.952931 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.952934 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.952935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.953009 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.953301 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.953303 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.953304 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.953481 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:21.953556 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:21.954129 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 19:03:21.954276 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 19:03:21.955427 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.941927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34759 (local address 127.0.0.1:57786)
0810 19:03:21.942466 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.942526 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:21.943973 (+ 1447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:21.944528 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:21.944538 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:21.944864 (+ 326us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:21.945310 (+ 446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.945322 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.946458 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.946461 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.947474 (+ 1013us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.947484 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.948207 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.948210 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.948342 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.948948 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:21.948964 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:21.950603 (+ 1639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:21.952534 (+ 1931us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.952543 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.952559 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.952844 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.953091 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.953093 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.953094 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.953221 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.953593 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:21.953598 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:21.953840 (+ 242us) client_negotiation.cc:769] Sending connection context
0810 19:03:21.954050 (+ 210us) client_negotiation.cc:240] Negotiation successful
0810 19:03:21.954345 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"thread_start_us":114,"threads_started":1}
I20260810 19:03:21.956950 31553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:21.961382 31447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } attempt: 3
Aug 10 19:03:21 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
Aug 10 19:03:21 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
I20260810 19:03:21.968504 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.957440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33198)
0810 19:03:21.957557 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.957571 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:21.957645 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:21.958410 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:21.958413 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:21.958540 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:21.958621 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.958626 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.960196 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.960202 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.961220 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.961236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.962100 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.962107 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.962214 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.962944 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:21.962962 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:21.964199 (+ 1237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:21.966688 (+ 2489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.966694 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.966696 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.966974 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.967343 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.967346 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.967347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.967432 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.968012 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:21.968015 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:21.968101 (+ 86us) client_negotiation.cc:769] Sending connection context
0810 19:03:21.968216 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 19:03:21.968346 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 19:03:21.968829 31563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.957612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33198 (local address 127.27.147.189:42945)
0810 19:03:21.957979 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:21.957993 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:21.958019 (+ 26us) server_negotiation.cc:407] Connection header received
0810 19:03:21.958065 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:21.958074 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:21.958267 (+ 193us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:21.958420 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:21.958731 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.960044 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.961344 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.961863 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.961964 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.964354 (+ 2390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:21.964368 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:21.964387 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:21.964431 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:21.966580 (+ 2149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.967139 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.967147 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.967152 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.967241 (+ 89us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.967528 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.967531 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.967533 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.967897 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:21.968038 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:21.968353 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 19:03:21.968490 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":249,"thread_start_us":151,"threads_started":1}
I20260810 19:03:21.970809 31553 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:21.971678 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.971346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50758)
0810 19:03:21.971507 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.971586 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260810 19:03:21.971910 31482 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
I20260810 19:03:21.972084 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.961891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33204 (local address 127.27.147.189:42945)
0810 19:03:21.962306 (+ 415us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:21.962315 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:21.962329 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:03:21.962366 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:21.962371 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:21.962412 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:21.962531 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:21.962973 (+ 442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.963854 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.964848 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.965147 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.965225 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.967284 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:21.967288 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:21.967290 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:21.967319 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:21.970204 (+ 2885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.970691 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.970694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.970695 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.970809 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:21.971159 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:21.971165 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:21.971170 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:21.971345 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:21.971526 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:21.971732 (+ 206us) server_negotiation.cc:299] Negotiation successful
0810 19:03:21.971846 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":82,"threads_started":1}
I20260810 19:03:21.972687 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.961803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33204)
0810 19:03:21.961988 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.962002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:21.962074 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:21.962548 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:21.962551 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:21.962709 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:21.962800 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.962805 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.964029 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.964033 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.964720 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:21.964732 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:21.965250 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:21.965253 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:21.965333 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:21.965758 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:21.965776 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:21.967137 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:21.970317 (+ 3180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.970322 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.970324 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.970588 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.970967 (+ 379us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:21.970971 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:21.970973 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:21.971063 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:21.971484 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:21.971487 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:21.971567 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 19:03:21.971649 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 19:03:21.972516 (+ 867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 19:03:21.972981 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.972639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50772)
0810 19:03:21.972762 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.972924 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:03:21.973099 31553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:33071: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.27.147.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab.
Entry for principal kudu/127.27.147.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab.
I20260810 19:03:21.973311 31447 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:21.974002 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.973767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50786)
0810 19:03:21.973878 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.973930 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260810 19:03:21.974561 28238 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
/tmp/dist-test-taskfzwUfv/build/debug/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:33071
--webserver_interface=127.27.147.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab
--principal=kudu/127.27.147.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.27.147.148:32893
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.27.147.190:34759,127.27.147.189:42945,127.27.147.188:33071
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 19:03:21.982146 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:21.981791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50788)
0810 19:03:21.981997 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:21.982066 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260810 19:03:21.982318 31447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:33071: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
I20260810 19:03:22.013478 31447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } attempt: 1
I20260810 19:03:22.014247 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.013980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50804)
0810 19:03:22.014118 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.014174 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:03:22.015129 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.014889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50820)
0810 19:03:22.014994 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.015046 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260810 19:03:22.015645 31447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:33071: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
I20260810 19:03:22.022337 31553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } attempt: 1
I20260810 19:03:22.023085 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.022743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50822)
0810 19:03:22.022936 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.022996 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 19:03:22.023893 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.023610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50836)
0810 19:03:22.023747 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.023813 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20260810 19:03:22.024071 31553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:33071: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
I20260810 19:03:22.090845 31447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } attempt: 2
I20260810 19:03:22.091719 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.091420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50844)
0810 19:03:22.091567 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.091639 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
I20260810 19:03:22.092294 31553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } attempt: 2
I20260810 19:03:22.092576 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.092332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50854)
0810 19:03:22.092461 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.092513 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 19:03:22.092895 31447 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:33071: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
I20260810 19:03:22.092960 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.092697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50860)
0810 19:03:22.092835 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.092892 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260810 19:03:22.093572 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.093349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50868)
0810 19:03:22.093461 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.093513 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260810 19:03:22.093719 31553 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.27.147.188:33071: Network error: Client connection negotiation failed: client connection to 127.27.147.188:33071: connect: Connection refused (error 111)
W20260810 19:03:22.093575 31566 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 19:03:22.093890 31566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 19:03:22.093919 31566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 19:03:22.093954 31566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 19:03:22.097899 31566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 19:03:22.097946 31566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 19:03:22.097965 31566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 19:03:22.097981 31566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 19:03:22.097996 31566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 19:03:22.103065 31566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.27.147.148:32893
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.27.147.190:34759,127.27.147.189:42945,127.27.147.188:33071
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/krb5kdc/kudu_127.27.147.188.keytab
--principal=kudu/127.27.147.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.27.147.188:33071
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.27.147.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type DEBUG
built by systest at 10 Aug 2026 18:13:35 UTC on ccycloud.quasar-witmzp.root.comops.site
build id 4578678
I20260810 19:03:22.103971 31566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 19:03:22.104871 31566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 19:03:22.113220 31575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 19:03:22.113231 31571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:22.113339 31566 server_base.cc:1034] running on GCE node
W20260810 19:03:22.113240 31580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 19:03:22.114108 31566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 19:03:22.115193 31566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 19:03:22.116382 31566 hybrid_clock.cc:648] HybridClock initialized: now 1786388602116367 us; error 24 us; skew 500 ppm
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388602, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 19:03:22.119071 31566 init.cc:377] Logged in from keytab as kudu/127.27.147.188@KRBTEST.COM (short username kudu)
I20260810 19:03:22.119899 31566 webserver.cc:460] Webserver started at http://127.27.147.188:40013/ using document root <none> and password file <none>
I20260810 19:03:22.120391 31566 fs_manager.cc:359] Metadata directory not provided
I20260810 19:03:22.120450 31566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 19:03:22.120602 31566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 19:03:22.122473 31566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data/instance:
uuid: "5f2ce32e11a94194905fa9dbd7052c0e"
format_stamp: "Formatted at 2026-08-10 19:03:22 on dist-test-slave-znh6"
server_key: "d6b707c8f43371a487e2caac5ffa6126"
server_key_iv: "57c75f13918ee4fd0cc0f971735b1277"
server_key_version: "encryptionkey@0"
I20260810 19:03:22.123082 31566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/wal/instance:
uuid: "5f2ce32e11a94194905fa9dbd7052c0e"
format_stamp: "Formatted at 2026-08-10 19:03:22 on dist-test-slave-znh6"
server_key: "d6b707c8f43371a487e2caac5ffa6126"
server_key_iv: "57c75f13918ee4fd0cc0f971735b1277"
server_key_version: "encryptionkey@0"
I20260810 19:03:22.126600 31566 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.000s sys 0.002s
I20260810 19:03:22.128918 31588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 19:03:22.130055 31566 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 19:03:22.130152 31566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data,/tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/wal
uuid: "5f2ce32e11a94194905fa9dbd7052c0e"
format_stamp: "Formatted at 2026-08-10 19:03:22 on dist-test-slave-znh6"
server_key: "d6b707c8f43371a487e2caac5ffa6126"
server_key_iv: "57c75f13918ee4fd0cc0f971735b1277"
server_key_version: "encryptionkey@0"
I20260810 19:03:22.130231 31566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 19:03:22.154090 31566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 19:03:22.157087 31566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 19:03:22.157243 31566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 19:03:22.164309 31566 rpc_server.cc:307] RPC server started. Bound to: 127.27.147.188:33071
I20260810 19:03:22.164590 31665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.27.147.188:33071 every 8 connection(s)
I20260810 19:03:22.165292 31566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/data/info.pb
I20260810 19:03:22.166795 28238 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu as pid 31566
I20260810 19:03:22.166962 28238 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskfzwUfv/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786388305279399-28238-0/minicluster-data/master-2/wal/instance
I20260810 19:03:22.167289 28238 external_mini_cluster.cc:1428] Setting key fc9d2de2de195b8eadc8e08675d04b0c
I20260810 19:03:22.168547 31666 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 19:03:22.172389 31666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388602, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
I20260810 19:03:22.181524 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.168753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50870 (local address 127.27.147.188:33071)
0810 19:03:22.169187 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.169201 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.169237 (+ 36us) server_negotiation.cc:407] Connection header received
0810 19:03:22.169994 (+ 757us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.170013 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.170372 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.170690 (+ 318us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.171516 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.172512 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.173434 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.173928 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.174091 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.176854 (+ 2763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.176859 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.176868 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.176901 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.178696 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.179180 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.179186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.179192 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.179346 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.179637 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.179639 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.179640 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.180005 (+ 365us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.180214 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.180499 (+ 285us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.180775 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":301,"thread_start_us":132,"threads_started":1}
I20260810 19:03:22.183547 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.173929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.190:34759 (local address 127.0.0.1:57788)
0810 19:03:22.174611 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.174661 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:22.174773 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:22.175123 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:22.175127 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:22.175441 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:22.175561 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.175566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.176351 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.176356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.177157 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.177167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.177632 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.177636 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.177758 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.178178 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:22.178189 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:22.179752 (+ 1563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:22.181536 (+ 1784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.181546 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.181551 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.181821 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.182107 (+ 286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.182110 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.182112 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.182199 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.182907 (+ 708us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:22.182913 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:22.183033 (+ 120us) client_negotiation.cc:769] Sending connection context
0810 19:03:22.183176 (+ 143us) client_negotiation.cc:240] Negotiation successful
0810 19:03:22.183383 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":173,"threads_started":1}
I20260810 19:03:22.183617 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.173996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57788 (local address 127.27.147.190:34759)
0810 19:03:22.174126 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.174131 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.174754 (+ 623us) server_negotiation.cc:407] Connection header received
0810 19:03:22.174914 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.174917 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.174976 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.175072 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.175691 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.176193 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.177279 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.177515 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.177629 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.179896 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.179899 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.179901 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.179923 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.181428 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.181926 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.181932 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.181936 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.182014 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.182290 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.182294 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.182297 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.182496 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.182578 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.183325 (+ 747us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.183444 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:03:22.184787 31666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } has no permanent_uuid. Determining permanent_uuid...
I20260810 19:03:22.184870 31553 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } attempt: 3
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.190@KRBTEST.COM
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388602, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.189@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:03:22.192057 31447 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } attempt: 3
I20260810 19:03:22.194317 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.183990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57790 (local address 127.27.147.190:34759)
0810 19:03:22.184110 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.184114 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.184127 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:03:22.184157 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.184160 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.184202 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.184279 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.185007 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.185706 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.186672 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.186972 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.187079 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.189759 (+ 2680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.189763 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.189765 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.189791 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.191621 (+ 1830us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.192256 (+ 635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.192260 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.192262 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.192356 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.192923 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.192926 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.192928 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.193167 (+ 239us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.193392 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.193995 (+ 603us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.194114 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 19:03:22.194340 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.185440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33208 (local address 127.27.147.189:42945)
0810 19:03:22.185577 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.185582 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.185596 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:03:22.185635 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.185637 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.185678 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.185766 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.186190 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.187008 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.187946 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.188211 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.188318 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.190312 (+ 1994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.190316 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.190319 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.190345 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.192305 (+ 1960us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.192804 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.192811 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.192815 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.192916 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.193255 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.193259 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.193261 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.193538 (+ 277us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.193609 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.194034 (+ 425us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.194172 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260810 19:03:22.194617 31556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.185283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50882)
0810 19:03:22.185565 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.185576 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:22.185635 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:22.185953 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:22.185956 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:22.186138 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:22.186223 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.186229 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.187207 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.187212 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.188059 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.188073 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.188773 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.188777 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.188888 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.189312 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:22.189327 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:22.191171 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:22.193070 (+ 1899us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.193073 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.193076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.193320 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.193636 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.193642 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.193646 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.193737 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.194161 (+ 424us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:22.194163 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:22.194244 (+ 81us) client_negotiation.cc:769] Sending connection context
0810 19:03:22.194321 (+ 77us) client_negotiation.cc:240] Negotiation successful
0810 19:03:22.194473 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":202}
I20260810 19:03:22.195508 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.185241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.189:42945 (local address 127.0.0.1:33208)
0810 19:03:22.185354 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.185366 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:22.185448 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:22.185798 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:22.185802 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:22.185972 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:22.186079 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.186084 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.187131 (+ 1047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.187134 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.187817 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.187826 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.188332 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.188336 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.188440 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.188881 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:22.188901 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:22.190176 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:22.192440 (+ 2264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.192444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.192447 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.192682 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.193038 (+ 356us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.193044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.193049 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.193141 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.193684 (+ 543us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:22.193689 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:22.193837 (+ 148us) client_negotiation.cc:769] Sending connection context
0810 19:03:22.195212 (+ 1375us) client_negotiation.cc:240] Negotiation successful
0810 19:03:22.195343 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":23}
I20260810 19:03:22.195590 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.185346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.27.147.188:33071)
0810 19:03:22.185775 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.185779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.185789 (+ 10us) server_negotiation.cc:407] Connection header received
0810 19:03:22.185813 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.185816 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.185854 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.185941 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.186338 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.187075 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.188222 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.188632 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.189093 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.191322 (+ 2229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.191326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.191328 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.191360 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.192962 (+ 1602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.193453 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.193456 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.193457 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.193530 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.193851 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.193854 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.193856 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.194054 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.195042 (+ 988us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.195117 (+ 75us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.195434 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":340}
I20260810 19:03:22.196404 31666 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.190@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
I20260810 19:03:22.200492 31553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5: Bootstrap starting.
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388602, etypes {rep=17 tkt=17 ses=17}, kudu/127.27.147.188@KRBTEST.COM for kudu/127.27.147.188@KRBTEST.COM
Aug 10 19:03:22 dist-test-slave-znh6 krb5kdc[31331](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786388601, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.27.147.189@KRBTEST.COM
I20260810 19:03:22.203506 31553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:22.203848 31453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.192430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50892)
0810 19:03:22.192517 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.192530 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:22.192596 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:22.193215 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:22.193221 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:22.193392 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:22.193479 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.193485 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.195203 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.195208 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.195963 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.195977 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.196683 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.196687 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.196774 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.197242 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:22.197256 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:22.198616 (+ 1360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:22.201160 (+ 2544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.201167 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.201171 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.201451 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.202716 (+ 1265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.202719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.202721 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.202845 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.203297 (+ 452us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:22.203302 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:22.203428 (+ 126us) client_negotiation.cc:769] Sending connection context
0810 19:03:22.203548 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 19:03:22.203690 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260810 19:03:22.204609 31674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.192647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50892 (local address 127.27.147.188:33071)
0810 19:03:22.192976 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.192988 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.193000 (+ 12us) server_negotiation.cc:407] Connection header received
0810 19:03:22.193032 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.193035 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.193074 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.193592 (+ 518us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.193691 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.195112 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.196101 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.196539 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.198749 (+ 2210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.198881 (+ 132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.198885 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.198887 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.198905 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.201007 (+ 2102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.202141 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.202145 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.202146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.202231 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.202970 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.202973 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.202974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.203168 (+ 194us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.203284 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.203662 (+ 378us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.203768 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":263,"thread_start_us":95,"threads_started":1}
I20260810 19:03:22.204643 31553 log.cc:824] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:22.206725 31670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.196942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.27.147.188:33071 (local address 127.0.0.1:50896)
0810 19:03:22.197051 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0810 19:03:22.197065 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 19:03:22.197135 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 19:03:22.197704 (+ 569us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 19:03:22.197706 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 19:03:22.197867 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0810 19:03:22.197924 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.197929 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.198657 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.198664 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.199418 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 19:03:22.199428 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.200035 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.200041 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 19:03:22.200113 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.200509 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 19:03:22.200523 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0810 19:03:22.202569 (+ 2046us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 19:03:22.205283 (+ 2714us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.205285 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.205287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.205528 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.205804 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 19:03:22.205807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 19:03:22.205809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 19:03:22.205897 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 19:03:22.206280 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 19:03:22.206283 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 19:03:22.206359 (+ 76us) client_negotiation.cc:769] Sending connection context
0810 19:03:22.206447 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 19:03:22.206567 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":50}
I20260810 19:03:22.207271 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.197091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50896 (local address 127.27.147.188:33071)
0810 19:03:22.197517 (+ 426us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.197529 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.197546 (+ 17us) server_negotiation.cc:407] Connection header received
0810 19:03:22.197579 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.197581 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.197623 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.197694 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.198028 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.198557 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.199533 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.199841 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.199931 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.202715 (+ 2784us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.202722 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.202727 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.202757 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.205167 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.205630 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.205633 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.205634 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.205710 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.205997 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.206000 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.206002 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.206171 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.206250 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.207054 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.207152 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260810 19:03:22.207352 31553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5: No bootstrap required, opened a new log
I20260810 19:03:22.207751 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.195728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33216 (local address 127.27.147.189:42945)
0810 19:03:22.195860 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.195864 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.195909 (+ 45us) server_negotiation.cc:407] Connection header received
0810 19:03:22.196017 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.196020 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.196062 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.196140 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.196865 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.199013 (+ 2148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.200066 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.200422 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.200514 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.203877 (+ 3363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.203880 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.203884 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.203908 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.205616 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.206111 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.206114 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.206116 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.206198 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.206498 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.206501 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.206502 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.206707 (+ 205us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.206909 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.207500 (+ 591us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.207602 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:03:22.207886 31447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47: Bootstrap starting.
I20260810 19:03:22.210040 31447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:22.210539 31553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.210702 31553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 62bddad7833a4b23a66363b0f42048d5, State: Initialized, Role: FOLLOWER
I20260810 19:03:22.210672 31447 log.cc:824] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:22.211055 31553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.212404 31553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:22.212448 31447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47: No bootstrap required, opened a new log
I20260810 19:03:22.213418 31666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: Bootstrap starting.
I20260810 19:03:22.214975 31692 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 19:03:22.216113 31666 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: Neither blocks nor log segments found. Creating new log.
I20260810 19:03:22.216068 31447 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.216267 31447 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3e6c577f85a4defb2e04191d656af47, State: Initialized, Role: FOLLOWER
I20260810 19:03:22.216586 31447 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.216924 31666 log.cc:824] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: Log is configured to *not* fsync() on all Append() calls
I20260810 19:03:22.217463 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.217602 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:22.217881 31447 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:22.219056 31666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: No bootstrap required, opened a new log
I20260810 19:03:22.220055 31693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.220186 31693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:22.220290 28238 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 19:03:22.217133 31457 master_runner.cc:186] Error getting master registration for 127.27.147.190:34759: OK, instance_id { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" instance_seqno: 1786388601713445 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 19:03:22.220770 31457 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 19:03:22.221679 31405 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "f3e6c577f85a4defb2e04191d656af47"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57790
I20260810 19:03:22.221899 31405 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 19:03:22.222010 31405 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:22.223124 31666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.223336 31405 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.223393 31666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f2ce32e11a94194905fa9dbd7052c0e, State: Initialized, Role: FOLLOWER
I20260810 19:03:22.223685 31666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.224258 31405 leader_election.cc:288] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [CANDIDATE]: Term 1 election: Requested vote from peers 62bddad7833a4b23a66363b0f42048d5 (127.27.147.189:42945), 5f2ce32e11a94194905fa9dbd7052c0e (127.27.147.188:33071)
W20260810 19:03:22.224705 31715 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 19:03:22.224783 31715 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:03:22.224936 31666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 19:03:22.225224 31518 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3e6c577f85a4defb2e04191d656af47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "62bddad7833a4b23a66363b0f42048d5"
I20260810 19:03:22.225477 31518 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:22.226944 31518 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e6c577f85a4defb2e04191d656af47 in term 1.
I20260810 19:03:22.227533 31378 leader_election.cc:302] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 62bddad7833a4b23a66363b0f42048d5, f3e6c577f85a4defb2e04191d656af47; no voters:
I20260810 19:03:22.227314 31627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f3e6c577f85a4defb2e04191d656af47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "5f2ce32e11a94194905fa9dbd7052c0e"
I20260810 19:03:22.227792 31693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 19:03:22.227929 31627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [term 0 FOLLOWER]: Advancing to term 1
I20260810 19:03:22.228032 31693 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [term 1 LEADER]: Becoming Leader. State: Replica: f3e6c577f85a4defb2e04191d656af47, State: Running, Role: LEADER
I20260810 19:03:22.228344 31693 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } }
I20260810 19:03:22.229120 31627 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f3e6c577f85a4defb2e04191d656af47 in term 1.
I20260810 19:03:22.229753 31719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.229943 31719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:22.230157 31693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e6c577f85a4defb2e04191d656af47. Latest consensus state: current_term: 1 leader_uuid: "f3e6c577f85a4defb2e04191d656af47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.230250 31693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:22.239095 31746 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260810 19:03:22.240144 31734 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260810 19:03:22.240182 31746 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260810 19:03:22.240226 31734 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 19:03:22.240317 31564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.229314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33228 (local address 127.27.147.189:42945)
0810 19:03:22.229446 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.229451 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.230634 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 19:03:22.230741 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.230745 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.230800 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.230903 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.231932 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.232699 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.233904 (+ 1205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.234233 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.234311 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.236253 (+ 1942us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.236259 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.236263 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.236289 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.238233 (+ 1944us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.238761 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.238765 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.238767 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.238883 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.239311 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.239314 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.239315 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.239549 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.239654 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.240038 (+ 384us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.240160 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260810 19:03:22.240710 31450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.229987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57798 (local address 127.27.147.190:34759)
0810 19:03:22.230088 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.230095 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.230967 (+ 872us) server_negotiation.cc:407] Connection header received
0810 19:03:22.231144 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.231148 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.231192 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.231240 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.232139 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.232833 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.234018 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.234328 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.234445 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.237052 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.237056 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.237058 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.237083 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.238766 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.239309 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.239312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.239314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.239416 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.239742 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.239746 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.239748 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.239959 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.240065 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.240457 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.240576 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 19:03:22.246218 31627 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [term 1 FOLLOWER]: Refusing update from remote peer f3e6c577f85a4defb2e04191d656af47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:03:22.247073 31518 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [term 1 FOLLOWER]: Refusing update from remote peer f3e6c577f85a4defb2e04191d656af47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 19:03:22.247879 31693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:03:22.248296 31693 consensus_queue.cc:1035] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 19:03:22.252234 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e6c577f85a4defb2e04191d656af47. Latest consensus state: current_term: 1 leader_uuid: "f3e6c577f85a4defb2e04191d656af47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.252362 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:22.253362 31669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:22.243844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50900 (local address 127.27.147.188:33071)
0810 19:03:22.243963 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.243967 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.243982 (+ 15us) server_negotiation.cc:407] Connection header received
0810 19:03:22.244124 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.244127 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.244174 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.244225 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.245106 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.245885 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.247126 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:22.247492 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:22.247587 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:22.249660 (+ 2073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 19:03:22.249664 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 19:03:22.249667 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 19:03:22.249693 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 19:03:22.251442 (+ 1749us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.251936 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.251940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.251942 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.252023 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 19:03:22.252309 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 19:03:22.252312 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 19:03:22.252314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 19:03:22.252638 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 19:03:22.252739 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:22.253081 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 19:03:22.253198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 19:03:22.256610 31719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3e6c577f85a4defb2e04191d656af47. Latest consensus state: current_term: 1 leader_uuid: "f3e6c577f85a4defb2e04191d656af47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.256719 31719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:22.259618 31693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e6c577f85a4defb2e04191d656af47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.259734 31693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:22.260684 31680 sys_catalog.cc:455] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e6c577f85a4defb2e04191d656af47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.260802 31680 sys_catalog.cc:458] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:22.261960 31693 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f3e6c577f85a4defb2e04191d656af47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.262035 31750 mvcc.cc:205] Tried to move back new op lower bound from 7317047714791571456 to 7317047714730819584. Current Snapshot: MvccSnapshot[applied={T|T < 7317047714791571456}]
I20260810 19:03:22.262071 31693 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47 [sys.catalog]: This master's current role is: LEADER
I20260810 19:03:22.264452 31719 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f3e6c577f85a4defb2e04191d656af47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3e6c577f85a4defb2e04191d656af47" member_type: VOTER last_known_addr { host: "127.27.147.190" port: 34759 } } peers { permanent_uuid: "62bddad7833a4b23a66363b0f42048d5" member_type: VOTER last_known_addr { host: "127.27.147.189" port: 42945 } } peers { permanent_uuid: "5f2ce32e11a94194905fa9dbd7052c0e" member_type: VOTER last_known_addr { host: "127.27.147.188" port: 33071 } } }
I20260810 19:03:22.264554 31719 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 19:03:22.264703 31752 mvcc.cc:205] Tried to move back new op lower bound from 7317047714791571456 to 7317047714730819584. Current Snapshot: MvccSnapshot[applied={T|T < 7317047714791571456}]
I20260810 19:03:22.267952 31746 catalog_manager.cc:1347] Generated new cluster ID: f0ec7bc2f56f41c385a39d585553a5a7
I20260810 19:03:22.268023 31746 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 19:03:22.308445 31746 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 19:03:22.309834 31746 catalog_manager.cc:1504] Loading token signing keys...
I20260810 19:03:22.319195 31746 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3e6c577f85a4defb2e04191d656af47: Generated new TSK 0
I20260810 19:03:22.320055 31746 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 19:03:22.357291 31450 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.356092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57802 (local address 127.27.147.190:34759)
0810 19:03:22.356217 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.356222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.356236 (+ 14us) server_negotiation.cc:407] Connection header received
0810 19:03:22.356283 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.356286 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.356343 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.356442 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.357150 (+ 708us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57802: BlockingRecv error: recv got EOF from 127.0.0.1:57802 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 19:03:22.357821 31669 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.355733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50916 (local address 127.27.147.188:33071)
0810 19:03:22.355856 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.355862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.356744 (+ 882us) server_negotiation.cc:407] Connection header received
0810 19:03:22.356927 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.356930 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.356985 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.357037 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.357706 (+ 669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50916: BlockingRecv error: recv got EOF from 127.0.0.1:50916 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260810 19:03:22.358562 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.355327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33240 (local address 127.27.147.189:42945)
0810 19:03:22.356001 (+ 674us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.356006 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.356215 (+ 209us) server_negotiation.cc:407] Connection header received
0810 19:03:22.356387 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.356390 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.356442 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.356494 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.358405 (+ 1911us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33240: BlockingRecv error: recv got EOF from 127.0.0.1:33240 (error 108)
Metrics: {"server-negotiator.queue_time_us":579}
W20260810 19:03:22.361950 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.360846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33248 (local address 127.27.147.189:42945)
0810 19:03:22.360953 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.360958 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.361131 (+ 173us) server_negotiation.cc:407] Connection header received
0810 19:03:22.361262 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.361265 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.361339 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.361434 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.361854 (+ 420us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33248: BlockingRecv error: recv got EOF from 127.0.0.1:33248 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260810 19:03:22.365964 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.365147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33262 (local address 127.27.147.189:42945)
0810 19:03:22.365256 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.365260 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.365273 (+ 13us) server_negotiation.cc:407] Connection header received
0810 19:03:22.365315 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.365318 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.365359 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.365447 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.365870 (+ 423us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33262: BlockingRecv error: recv got EOF from 127.0.0.1:33262 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 19:03:22.371081 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.369934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33270 (local address 127.27.147.189:42945)
0810 19:03:22.370057 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.370064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.370254 (+ 190us) server_negotiation.cc:407] Connection header received
0810 19:03:22.370435 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.370439 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.370491 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.370578 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.370987 (+ 409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33270: BlockingRecv error: recv got EOF from 127.0.0.1:33270 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
W20260810 19:03:22.378239 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.377039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33284 (local address 127.27.147.189:42945)
0810 19:03:22.377185 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.377190 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.377381 (+ 191us) server_negotiation.cc:407] Connection header received
0810 19:03:22.377525 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.377529 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.377582 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.377668 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.378144 (+ 476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33284: BlockingRecv error: recv got EOF from 127.0.0.1:33284 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
W20260810 19:03:22.389541 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.388291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33300 (local address 127.27.147.189:42945)
0810 19:03:22.388397 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.388401 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.388665 (+ 264us) server_negotiation.cc:407] Connection header received
0810 19:03:22.388790 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.388793 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.388847 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.388940 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.389427 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33300: BlockingRecv error: recv got EOF from 127.0.0.1:33300 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260810 19:03:22.409293 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.407783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33308 (local address 127.27.147.189:42945)
0810 19:03:22.407902 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.407907 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.408192 (+ 285us) server_negotiation.cc:407] Connection header received
0810 19:03:22.408296 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.408300 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.408415 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.408503 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.409174 (+ 671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33308: BlockingRecv error: recv got EOF from 127.0.0.1:33308 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
W20260810 19:03:22.444950 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.443894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33316 (local address 127.27.147.189:42945)
0810 19:03:22.444048 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.444052 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.444204 (+ 152us) server_negotiation.cc:407] Connection header received
0810 19:03:22.444328 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.444331 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.444385 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.444465 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.444854 (+ 389us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33316: BlockingRecv error: recv got EOF from 127.0.0.1:33316 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20260810 19:03:22.512547 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.511303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33322 (local address 127.27.147.189:42945)
0810 19:03:22.511505 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.511509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.511629 (+ 120us) server_negotiation.cc:407] Connection header received
0810 19:03:22.511720 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.511723 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.511778 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.511946 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.512445 (+ 499us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33322: BlockingRecv error: recv got EOF from 127.0.0.1:33322 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 19:03:22.644090 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.642780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33330 (local address 127.27.147.189:42945)
0810 19:03:22.642957 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.642962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.643122 (+ 160us) server_negotiation.cc:407] Connection header received
0810 19:03:22.643320 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.643324 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.643388 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.643485 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.643966 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33330: BlockingRecv error: recv got EOF from 127.0.0.1:33330 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260810 19:03:22.903669 31564 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 19:03:22.902377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33344 (local address 127.27.147.189:42945)
0810 19:03:22.902532 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:22.902538 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:22.902632 (+ 94us) server_negotiation.cc:407] Connection header received
0810 19:03:22.902846 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:22.902849 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:22.902924 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:22.903031 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 19:03:22.903546 (+ 515us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33344: BlockingRecv error: recv got EOF from 127.0.0.1:33344 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 19:03:23.218246 31692 catalog_manager.cc:1259] Loaded cluster ID: f0ec7bc2f56f41c385a39d585553a5a7
I20260810 19:03:23.218364 31692 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5: loading cluster ID for follower catalog manager: success
I20260810 19:03:23.220244 31692 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5: acquiring CA information for follower catalog manager: success
I20260810 19:03:23.221403 31692 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 62bddad7833a4b23a66363b0f42048d5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:03:23.241613 31734 catalog_manager.cc:1259] Loaded cluster ID: f0ec7bc2f56f41c385a39d585553a5a7
I20260810 19:03:23.241710 31734 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: loading cluster ID for follower catalog manager: success
I20260810 19:03:23.243980 31734 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: acquiring CA information for follower catalog manager: success
I20260810 19:03:23.245343 31734 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 5f2ce32e11a94194905fa9dbd7052c0e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 19:03:23.422379 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:23.418077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33358 (local address 127.27.147.189:42945)
0810 19:03:23.418479 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:23.418486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:23.418509 (+ 23us) server_negotiation.cc:407] Connection header received
0810 19:03:23.418578 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:23.418581 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:23.418727 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:23.418852 (+ 125us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:23.419140 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.419785 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.420715 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.420955 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.421044 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:23.421386 (+ 342us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:23.421841 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:23.421936 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:23.422091 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 19:03:23.422198 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":244,"thread_start_us":78,"threads_started":1}
I20260810 19:03:23.429699 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:23.425836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50918 (local address 127.27.147.188:33071)
0810 19:03:23.426079 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:23.426102 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:23.426123 (+ 21us) server_negotiation.cc:407] Connection header received
0810 19:03:23.426223 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:23.426226 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:23.426379 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:23.426506 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:23.426729 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.427300 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.428075 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.428283 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.428357 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:23.428683 (+ 326us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:23.429121 (+ 438us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:23.429218 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:23.429445 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 19:03:23.429519 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167,"thread_start_us":66,"threads_started":1}
I20260810 19:03:23.436510 31864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:23.433020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50930 (local address 127.27.147.188:33071)
0810 19:03:23.433137 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:23.433141 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:23.433157 (+ 16us) server_negotiation.cc:407] Connection header received
0810 19:03:23.433305 (+ 148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:23.433312 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:23.433369 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:23.433418 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:23.433755 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.434456 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.435411 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.435635 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.435746 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:23.436033 (+ 287us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:23.436109 (+ 76us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:23.436179 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:23.436311 (+ 132us) server_negotiation.cc:299] Negotiation successful
0810 19:03:23.436368 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 19:03:23.437120 31857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:23.432799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33374 (local address 127.27.147.189:42945)
0810 19:03:23.432919 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:23.432924 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:23.433407 (+ 483us) server_negotiation.cc:407] Connection header received
0810 19:03:23.433653 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:23.433656 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:23.433710 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:23.433788 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:23.434013 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.434775 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.435909 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.436220 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.436315 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:23.436635 (+ 320us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:23.436739 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:23.436804 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:23.436945 (+ 141us) server_negotiation.cc:299] Negotiation successful
0810 19:03:23.437002 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 19:03:23.438031 31871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 19:03:23.432792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57814 (local address 127.27.147.190:34759)
0810 19:03:23.433285 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 19:03:23.433290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 19:03:23.433556 (+ 266us) server_negotiation.cc:407] Connection header received
0810 19:03:23.433959 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 19:03:23.433963 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 19:03:23.434111 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 19:03:23.434243 (+ 132us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 19:03:23.434484 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.435250 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.436356 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 19:03:23.436692 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 19:03:23.436783 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 19:03:23.437246 (+ 463us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 19:03:23.437609 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 19:03:23.437711 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 19:03:23.437824 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 19:03:23.437877 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":89,"threads_started":1}
I20260810 19:03:23.441170 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31350
I20260810 19:03:23.456001 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31457
I20260810 19:03:23.469679 28238 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskfzwUfv/build/debug/bin/kudu with pid 31566
2026-08-10T19:03:23Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (3357 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (6657 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (298189 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST